[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:37 dist-test-slave-6wvm krb5kdc[2045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:37 dist-test-slave-6wvm krb5kdc[2045](info): set up 2 sockets
Sep 24 16:11:37 dist-test-slave-6wvm krb5kdc[2045](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 16:11:38.330431 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:11:38 dist-test-slave-6wvm krb5kdc[2045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:11:38.458504 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:40525
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:40525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:38.651677 2058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:38.652005 2058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:38.652081 2058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:38.656232 2058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:38.656314 2058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:38.656353 2058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:38.656409 2058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:38.656450 2058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:38.661774 2058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:40525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:40525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:38.662963 2058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:38.664129 2058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:11:38.702900 2058 server_base.cc:1034] running on GCE node
W20260924 16:11:38.702869 2065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:38.703009 2063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:38.702927 2067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:38.703809 2058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:38.705181 2058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:38.706305 2058 hybrid_clock.cc:648] HybridClock initialized: now 1790266298706305 us; error 76 us; skew 500 ppm
Sep 24 16:11:38 dist-test-slave-6wvm krb5kdc[2045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266298, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:38.709589 2058 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:11:38.710637 2058 webserver.cc:460] Webserver started at http://127.1.253.126:43689/ using document root <none> and password file <none>
I20260924 16:11:38.711261 2058 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:38.711335 2058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:38.711565 2058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:38.713954 2058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "fed2f5ed279143308ea1b617e279bfa6"
format_stamp: "Formatted at 2026-09-24 16:11:38 on dist-test-slave-6wvm"
server_key: "cbca0ef2b2e5f862a8103ebbcdf5f58c"
server_key_iv: "5020180e49cedac9fcdd0f5bb3627cb1"
server_key_version: "encryptionkey@0"
I20260924 16:11:38.714668 2058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "fed2f5ed279143308ea1b617e279bfa6"
format_stamp: "Formatted at 2026-09-24 16:11:38 on dist-test-slave-6wvm"
server_key: "cbca0ef2b2e5f862a8103ebbcdf5f58c"
server_key_iv: "5020180e49cedac9fcdd0f5bb3627cb1"
server_key_version: "encryptionkey@0"
I20260924 16:11:38.719334 2058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:11:38.721906 2074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:38.723301 2058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:11:38.723632 2058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "fed2f5ed279143308ea1b617e279bfa6"
format_stamp: "Formatted at 2026-09-24 16:11:38 on dist-test-slave-6wvm"
server_key: "cbca0ef2b2e5f862a8103ebbcdf5f58c"
server_key_iv: "5020180e49cedac9fcdd0f5bb3627cb1"
server_key_version: "encryptionkey@0"
I20260924 16:11:38.723764 2058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:38.747270 2058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:38.750491 2058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:38.750732 2058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:38.766981 2058 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:40525
I20260924 16:11:38.766999 2127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:40525 every 8 connection(s)
I20260924 16:11:38.768224 2058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:38.771457 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 2058
I20260924 16:11:38.771589 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:11:38.772920 2037 external_mini_cluster.cc:1428] Setting key e1e024d898cfd248823a1491e7dfdfa6
I20260924 16:11:38.774557 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:38.783772 2128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6: Bootstrap starting.
I20260924 16:11:38.787137 2128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:38.787958 2128 log.cc:824] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:38.790024 2128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6: No bootstrap required, opened a new log
Sep 24 16:11:38 dist-test-slave-6wvm krb5kdc[2045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:38.794678 2128 raft_consensus.cc:348] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed2f5ed279143308ea1b617e279bfa6" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 40525 } }
I20260924 16:11:38.794948 2128 raft_consensus.cc:374] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:38.795030 2128 raft_consensus.cc:729] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed2f5ed279143308ea1b617e279bfa6, State: Initialized, Role: FOLLOWER
I20260924 16:11:38.795549 2128 consensus_queue.cc:260] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed2f5ed279143308ea1b617e279bfa6" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 40525 } }
I20260924 16:11:38.795710 2128 raft_consensus.cc:388] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:38.795800 2128 raft_consensus.cc:482] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:38.795953 2128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:38.797092 2128 raft_consensus.cc:504] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed2f5ed279143308ea1b617e279bfa6" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 40525 } }
I20260924 16:11:38.797495 2128 leader_election.cc:302] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fed2f5ed279143308ea1b617e279bfa6; no voters:
I20260924 16:11:38.797890 2128 leader_election.cc:288] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:38.797978 2133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:38.798225 2133 raft_consensus.cc:686] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [term 1 LEADER]: Becoming Leader. State: Replica: fed2f5ed279143308ea1b617e279bfa6, State: Running, Role: LEADER
I20260924 16:11:38.798410 2130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:38.777063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.1.253.126:40525)
0924 16:11:38.777583 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:38.777598 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:38.778643 (+ 1045us) server_negotiation.cc:407] Connection header received
0924 16:11:38.781770 (+ 3127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:38.781786 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:38.782161 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:38.782549 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:38.784852 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:38.785891 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:38.787174 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:38.787721 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:38.787892 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:38.791933 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:38.791978 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:38.791991 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:38.792019 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:38.794437 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:38.794998 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:38.795004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:38.795008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:38.795202 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:38.795566 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:38.795569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:38.795571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:38.796117 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:38.796335 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:38.796952 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 16:11:38.797266 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":145,"threads_started":1}
I20260924 16:11:38.798704 2133 consensus_queue.cc:237] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed2f5ed279143308ea1b617e279bfa6" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 40525 } }
I20260924 16:11:38.799326 2128 sys_catalog.cc:564] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:38.800235 2135 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fed2f5ed279143308ea1b617e279bfa6. Latest consensus state: current_term: 1 leader_uuid: "fed2f5ed279143308ea1b617e279bfa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed2f5ed279143308ea1b617e279bfa6" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 40525 } } }
I20260924 16:11:38.800369 2135 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:38.801133 2139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:38.805930 2139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:38.807003 2134 sys_catalog.cc:455] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fed2f5ed279143308ea1b617e279bfa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fed2f5ed279143308ea1b617e279bfa6" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 40525 } } }
I20260924 16:11:38.807123 2134 sys_catalog.cc:458] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:38.812408 2139 catalog_manager.cc:1347] Generated new cluster ID: 93e097eb310f4973b49b6d961bbdf0ae
I20260924 16:11:38.812467 2139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:38.819208 2139 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:38.820123 2139 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:38.829419 2139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fed2f5ed279143308ea1b617e279bfa6: Generated new TSK 0
I20260924 16:11:38.830148 2139 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:11:38.861388 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:40525
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:38.980861 2155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:38.981138 2155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:38.981187 2155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:38.985425 2155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:38.985510 2155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:38.985611 2155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:11:38.991099 2155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:40525
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:38.992200 2155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:38.993305 2155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:39.004123 2160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:39.004216 2161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:39.004352 2163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:39.005059 2155 server_base.cc:1034] running on GCE node
I20260924 16:11:39.005476 2155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:39.006117 2155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:39.007300 2155 hybrid_clock.cc:648] HybridClock initialized: now 1790266299007283 us; error 33 us; skew 500 ppm
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266299, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:39.010002 2155 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:11:39.010869 2155 webserver.cc:460] Webserver started at http://127.1.253.65:39567/ using document root <none> and password file <none>
I20260924 16:11:39.011432 2155 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:39.011524 2155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:39.011757 2155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:39.013770 2155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "6cd7d7b1efcd428e959190065d5a1d5a"
format_stamp: "Formatted at 2026-09-24 16:11:39 on dist-test-slave-6wvm"
server_key: "831f27f8a2973b495d2cb3907b9155b8"
server_key_iv: "dc38384f4ee46e102142351e3236a739"
server_key_version: "encryptionkey@0"
I20260924 16:11:39.014384 2155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "6cd7d7b1efcd428e959190065d5a1d5a"
format_stamp: "Formatted at 2026-09-24 16:11:39 on dist-test-slave-6wvm"
server_key: "831f27f8a2973b495d2cb3907b9155b8"
server_key_iv: "dc38384f4ee46e102142351e3236a739"
server_key_version: "encryptionkey@0"
I20260924 16:11:39.017982 2155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:11:39.020552 2171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:39.021747 2155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:11:39.021878 2155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "6cd7d7b1efcd428e959190065d5a1d5a"
format_stamp: "Formatted at 2026-09-24 16:11:39 on dist-test-slave-6wvm"
server_key: "831f27f8a2973b495d2cb3907b9155b8"
server_key_iv: "dc38384f4ee46e102142351e3236a739"
server_key_version: "encryptionkey@0"
I20260924 16:11:39.021986 2155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:39.038033 2155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:39.041607 2155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:39.041823 2155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:39.042529 2155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:39.043493 2155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:39.043566 2155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:39.043640 2155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:39.043684 2155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:39.056825 2155 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:39771
I20260924 16:11:39.056855 2287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:39771 every 8 connection(s)
I20260924 16:11:39.057876 2155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:11:39.058521 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 2155
I20260924 16:11:39.058642 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:11:39.058980 2037 external_mini_cluster.cc:1428] Setting key a9350dd288bd1163770699ba51bb7f92
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266299, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
I20260924 16:11:39.077270 2130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:39.060959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:41549 (local address 127.1.253.126:40525)
0924 16:11:39.061133 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:39.061139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:39.061536 (+ 397us) server_negotiation.cc:407] Connection header received
0924 16:11:39.064187 (+ 2651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:39.064195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:39.064261 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:39.064341 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:39.065793 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.066760 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.068726 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.069241 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.069385 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:39.072622 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:39.072625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:39.072627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:39.072648 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:39.074583 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:39.075283 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:39.075291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:39.075295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:39.075407 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:39.075882 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:39.075888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:39.075893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:39.076213 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:39.076342 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:39.076883 (+ 541us) server_negotiation.cc:299] Negotiation successful
0924 16:11:39.077015 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:11:39.078065 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:39.060453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:40525 (local address 127.1.253.65:41549)
0924 16:11:39.061385 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:39.061433 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:39.063872 (+ 2439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:39.064715 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:39.064722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:39.065100 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:39.065629 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:39.065644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.066916 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.066919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:39.068567 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:39.068581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.069433 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.069438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:39.069770 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:39.070693 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:39.070710 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:39.072445 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:39.074731 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:39.074738 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:39.074754 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:39.075155 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:39.075558 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:39.075563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:39.075565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:39.075762 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:39.076405 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:39.076411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:39.076648 (+ 237us) client_negotiation.cc:769] Sending connection context
0924 16:11:39.076841 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 16:11:39.077184 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":158,"threads_started":1}
I20260924 16:11:39.079387 2288 heartbeater.cc:347] Connected to a master server at 127.1.253.126:40525
I20260924 16:11:39.079649 2288 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:39.080241 2288 heartbeater.cc:511] Master 127.1.253.126:40525 requested a full tablet report, sending...
I20260924 16:11:39.082114 2092 ts_manager.cc:194] Registered new tserver with Master: 6cd7d7b1efcd428e959190065d5a1d5a (127.1.253.65:39771)
I20260924 16:11:39.083359 2092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:41549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:11:39.092880 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:40525
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:39.220712 2293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:39.220956 2293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:39.221030 2293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:39.225153 2293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:39.225214 2293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:39.225319 2293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:11:39.231093 2293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:40525
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:39.232172 2293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:39.233213 2293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:39.243708 2299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:39.243742 2298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:39.243718 2293 server_base.cc:1034] running on GCE node
W20260924 16:11:39.243741 2302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:39.244436 2293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:39.245121 2293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:39.246320 2293 hybrid_clock.cc:648] HybridClock initialized: now 1790266299246296 us; error 30 us; skew 500 ppm
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266299, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:39.249020 2293 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:11:39.249933 2293 webserver.cc:460] Webserver started at http://127.1.253.66:42665/ using document root <none> and password file <none>
I20260924 16:11:39.250582 2293 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:39.250644 2293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:39.250890 2293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:39.252975 2293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "213c28c4a6de46479443a9cba0bd87af"
format_stamp: "Formatted at 2026-09-24 16:11:39 on dist-test-slave-6wvm"
server_key: "52379678e09ed3363c29f9a5199008fd"
server_key_iv: "2926b8ec3ad75ec14565ce59d0bc53a0"
server_key_version: "encryptionkey@0"
I20260924 16:11:39.253630 2293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "213c28c4a6de46479443a9cba0bd87af"
format_stamp: "Formatted at 2026-09-24 16:11:39 on dist-test-slave-6wvm"
server_key: "52379678e09ed3363c29f9a5199008fd"
server_key_iv: "2926b8ec3ad75ec14565ce59d0bc53a0"
server_key_version: "encryptionkey@0"
I20260924 16:11:39.257529 2293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 16:11:39.260265 2311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:39.261538 2293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:11:39.261684 2293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "213c28c4a6de46479443a9cba0bd87af"
format_stamp: "Formatted at 2026-09-24 16:11:39 on dist-test-slave-6wvm"
server_key: "52379678e09ed3363c29f9a5199008fd"
server_key_iv: "2926b8ec3ad75ec14565ce59d0bc53a0"
server_key_version: "encryptionkey@0"
I20260924 16:11:39.261802 2293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:39.285344 2293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:39.288302 2293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:39.288520 2293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:39.289188 2293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:39.290072 2293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:39.290146 2293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:39.290220 2293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:39.290292 2293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:39.301265 2293 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:36779
I20260924 16:11:39.301292 2426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:36779 every 8 connection(s)
I20260924 16:11:39.302381 2293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:11:39.311275 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 2293
I20260924 16:11:39.311388 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:11:39.311721 2037 external_mini_cluster.cc:1428] Setting key 781dbc52cab4f91c1603d38f33ba22d7
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266299, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:39.318123 2130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:39.304357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:49919 (local address 127.1.253.126:40525)
0924 16:11:39.304498 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:39.304501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:39.307100 (+ 2599us) server_negotiation.cc:407] Connection header received
0924 16:11:39.308018 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:39.308022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:39.308078 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:39.308190 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:39.309374 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.309906 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.311152 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.311437 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.311560 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:39.314365 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:39.314368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:39.314370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:39.314393 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:39.315928 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:39.316475 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:39.316478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:39.316480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:39.316568 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:39.316929 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:39.316932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:39.316934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:39.317164 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:39.317272 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:39.317785 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 16:11:39.317922 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:11:39.318986 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:39.304753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:40525 (local address 127.1.253.66:49919)
0924 16:11:39.306933 (+ 2180us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:39.306979 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:39.307796 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:39.308361 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:39.308368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:39.308739 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:39.309226 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:39.309240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.310048 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.310052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:39.310982 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:39.311015 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.311612 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.311618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:39.311823 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:39.312565 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:39.312589 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:39.314148 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:39.316068 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:39.316075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:39.316091 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:39.316362 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:39.316664 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:39.316667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:39.316669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:39.316826 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:39.317270 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:39.317277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:39.317534 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 16:11:39.317729 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 16:11:39.318085 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1958,"thread_start_us":1795,"threads_started":1}
I20260924 16:11:39.320330 2427 heartbeater.cc:347] Connected to a master server at 127.1.253.126:40525
I20260924 16:11:39.320593 2427 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:39.321111 2427 heartbeater.cc:511] Master 127.1.253.126:40525 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:11:39.322208 2092 ts_manager.cc:194] Registered new tserver with Master: 213c28c4a6de46479443a9cba0bd87af (127.1.253.66:36779)
I20260924 16:11:39.322844 2092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:49919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:11:39.341333 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:40525
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:39.461597 2432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:39.461880 2432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:39.461982 2432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:39.466085 2432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:39.466159 2432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:39.466262 2432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:11:39.471549 2432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:40525
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:39.472636 2432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:39.473676 2432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:11:39.483727 2432 server_base.cc:1034] running on GCE node
W20260924 16:11:39.483678 2438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:39.483678 2437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:39.483735 2440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:39.484407 2432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:39.485064 2432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:39.486222 2432 hybrid_clock.cc:648] HybridClock initialized: now 1790266299486190 us; error 44 us; skew 500 ppm
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266299, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:39.489043 2432 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:11:39.489948 2432 webserver.cc:460] Webserver started at http://127.1.253.67:38201/ using document root <none> and password file <none>
I20260924 16:11:39.490581 2432 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:39.490669 2432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:39.490907 2432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:39.492991 2432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "5f63db4943824bca977c2ade0e6b6f63"
format_stamp: "Formatted at 2026-09-24 16:11:39 on dist-test-slave-6wvm"
server_key: "9c01f836bc0f11d0cbfa5ff35d457f2f"
server_key_iv: "85fdd465835596e04fda2e5997a2715d"
server_key_version: "encryptionkey@0"
I20260924 16:11:39.493580 2432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "5f63db4943824bca977c2ade0e6b6f63"
format_stamp: "Formatted at 2026-09-24 16:11:39 on dist-test-slave-6wvm"
server_key: "9c01f836bc0f11d0cbfa5ff35d457f2f"
server_key_iv: "85fdd465835596e04fda2e5997a2715d"
server_key_version: "encryptionkey@0"
I20260924 16:11:39.497385 2432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 16:11:39.499959 2448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:39.501286 2432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:11:39.501417 2432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "5f63db4943824bca977c2ade0e6b6f63"
format_stamp: "Formatted at 2026-09-24 16:11:39 on dist-test-slave-6wvm"
server_key: "9c01f836bc0f11d0cbfa5ff35d457f2f"
server_key_iv: "85fdd465835596e04fda2e5997a2715d"
server_key_version: "encryptionkey@0"
I20260924 16:11:39.501521 2432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:39.520750 2432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:39.524196 2432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:39.524399 2432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:39.525017 2432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:39.525902 2432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:39.525965 2432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:39.526033 2432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:39.526077 2432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:39.535831 2432 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:34997
I20260924 16:11:39.535856 2563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:34997 every 8 connection(s)
I20260924 16:11:39.536746 2432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:11:39.538365 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 2432
I20260924 16:11:39.538496 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:11:39.538805 2037 external_mini_cluster.cc:1428] Setting key b62bd21c96253bfae1d075d9776f5505
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266299, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:39.550400 2130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:39.539115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:50627 (local address 127.1.253.126:40525)
0924 16:11:39.539253 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:39.539257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:39.540307 (+ 1050us) server_negotiation.cc:407] Connection header received
0924 16:11:39.541685 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:39.541688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:39.541732 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:39.541796 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:39.543037 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.543557 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.544642 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.544844 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.544927 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:39.547328 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:39.547331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:39.547333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:39.547356 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:39.548594 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:39.549064 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:39.549067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:39.549068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:39.549138 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:39.549429 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:39.549431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:39.549432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:39.549596 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:39.549666 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:39.550124 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 16:11:39.550219 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:11:39.551266 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:39.539607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:40525 (local address 127.1.253.67:50627)
0924 16:11:39.540162 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:39.540218 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:39.541502 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:39.541975 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:39.541983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:39.542424 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:39.542911 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:39.542924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.543661 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.543663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:39.544535 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:39.544543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.544977 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.544980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:39.545113 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:39.545660 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:39.545677 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:39.547178 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:39.548700 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:39.548704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:39.548716 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:39.548980 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:39.549217 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:39.549219 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:39.549220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:39.549349 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:39.549680 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:39.549685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:39.549923 (+ 238us) client_negotiation.cc:769] Sending connection context
0924 16:11:39.550077 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 16:11:39.550391 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":169,"threads_started":1}
I20260924 16:11:39.552536 2564 heartbeater.cc:347] Connected to a master server at 127.1.253.126:40525
I20260924 16:11:39.552815 2564 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:39.553447 2564 heartbeater.cc:511] Master 127.1.253.126:40525 requested a full tablet report, sending...
I20260924 16:11:39.554490 2088 ts_manager.cc:194] Registered new tserver with Master: 5f63db4943824bca977c2ade0e6b6f63 (127.1.253.67:34997)
I20260924 16:11:39.555039 2088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:50627
I20260924 16:11:39.565191 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266299, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266299, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.253.65@KRBTEST.COM
I20260924 16:11:39.592485 2571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:39.583098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.1.253.65:39771)
0924 16:11:39.583457 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:39.583464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:39.583489 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:11:39.583543 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:39.583555 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:39.583752 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:39.583877 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:39.584564 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.585327 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.586207 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.586650 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.586737 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:39.589238 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:39.589247 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:39.589262 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:39.589289 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:39.590878 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:39.591282 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:39.591287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:39.591288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:39.591367 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:39.591604 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:39.591606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:39.591607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:39.591846 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:39.592040 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:39.592217 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 16:11:39.592343 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":128,"threads_started":1}
W20260924 16:11:39.593601 2201 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56612
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.65@KRBTEST.COM
I20260924 16:11:39.621712 2571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:39.612967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.1.253.65:39771)
0924 16:11:39.613138 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:39.613143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:39.613281 (+ 138us) server_negotiation.cc:407] Connection header received
0924 16:11:39.613475 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:39.613479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:39.613523 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:39.613618 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:39.614361 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.614876 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.615727 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:39.615949 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:39.616059 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:39.618413 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:39.618416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:39.618419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:39.618444 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:39.620081 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:39.620485 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:39.620489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:39.620490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:39.620572 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:39.620819 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:39.620822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:39.620823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:39.621188 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:39.621327 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:39.621486 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 16:11:39.621587 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:11:39.622953 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 2155
I20260924 16:11:39.630087 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 2293
I20260924 16:11:39.636988 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 2432
I20260924 16:11:39.643383 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 2058
2026-09-24T16:11:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2390 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2578](info): set up 2 sockets
Sep 24 16:11:39 dist-test-slave-6wvm krb5kdc[2578](info): commencing operation
krb5kdc: starting...
W20260924 16:11:40.685777 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:11:40 dist-test-slave-6wvm krb5kdc[2578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:11:40.809015 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:39329
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:41089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:39329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:40.923735 2592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:40.924031 2592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:40.924073 2592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:40.928205 2592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:40.928280 2592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:40.928300 2592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:40.928318 2592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:40.928333 2592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:40.933804 2592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:41089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:39329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:39329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:40.934990 2592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:40.935982 2592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:40.944662 2597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:40.944638 2598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:40.944723 2592 server_base.cc:1034] running on GCE node
W20260924 16:11:40.944639 2600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:40.945425 2592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:40.946596 2592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:40.947757 2592 hybrid_clock.cc:648] HybridClock initialized: now 1790266300947724 us; error 46 us; skew 500 ppm
Sep 24 16:11:40 dist-test-slave-6wvm krb5kdc[2578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266300, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:40.950531 2592 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:11:40.951519 2592 webserver.cc:460] Webserver started at http://127.1.253.126:43569/ using document root <none> and password file <none>
I20260924 16:11:40.952104 2592 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:40.952198 2592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:40.952442 2592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:40.954602 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "e9f7b39b2f24477db1f1f982089f9b62"
format_stamp: "Formatted at 2026-09-24 16:11:40 on dist-test-slave-6wvm"
server_key: "260ccdcdd504b88c28af5ace1bb7d8bb"
server_key_iv: "55e068d52665382b5cefc9fcde463741"
server_key_version: "encryptionkey@0"
I20260924 16:11:40.955173 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "e9f7b39b2f24477db1f1f982089f9b62"
format_stamp: "Formatted at 2026-09-24 16:11:40 on dist-test-slave-6wvm"
server_key: "260ccdcdd504b88c28af5ace1bb7d8bb"
server_key_iv: "55e068d52665382b5cefc9fcde463741"
server_key_version: "encryptionkey@0"
I20260924 16:11:40.958981 2592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:11:40.961333 2608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:40.962666 2592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:11:40.962803 2592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "e9f7b39b2f24477db1f1f982089f9b62"
format_stamp: "Formatted at 2026-09-24 16:11:40 on dist-test-slave-6wvm"
server_key: "260ccdcdd504b88c28af5ace1bb7d8bb"
server_key_iv: "55e068d52665382b5cefc9fcde463741"
server_key_version: "encryptionkey@0"
I20260924 16:11:40.962909 2592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:40.979523 2592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:40.982388 2592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:40.982595 2592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:40.989902 2592 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:39329
I20260924 16:11:40.989930 2661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:39329 every 8 connection(s)
I20260924 16:11:40.990872 2592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:40.993674 2662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:40.995442 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 2592
I20260924 16:11:40.995561 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:11:40.995844 2037 external_mini_cluster.cc:1428] Setting key 0c26e7e7ff2e92a6028570e4319df291
I20260924 16:11:41.000087 2662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62: Bootstrap starting.
I20260924 16:11:41.002393 2662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:41.003219 2662 log.cc:824] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:11:41 dist-test-slave-6wvm krb5kdc[2578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:41.005613 2662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62: No bootstrap required, opened a new log
I20260924 16:11:41.010092 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:40.997356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.1.253.126:39329)
0924 16:11:40.997796 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:40.997810 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:40.997846 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:11:40.998502 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:40.998520 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:40.998856 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:40.999188 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:41.000014 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.001041 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.001906 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.002459 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.002633 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.005574 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:41.005579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:41.005594 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:41.005621 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:41.007391 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.007851 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.007856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.007860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.007964 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.008202 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.008204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.008205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.008576 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:41.008711 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.009040 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.009365 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":139,"threads_started":1}
I20260924 16:11:41.011047 2662 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f7b39b2f24477db1f1f982089f9b62" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 39329 } }
I20260924 16:11:41.011341 2662 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:41.011401 2662 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f7b39b2f24477db1f1f982089f9b62, State: Initialized, Role: FOLLOWER
I20260924 16:11:41.011883 2662 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f7b39b2f24477db1f1f982089f9b62" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 39329 } }
I20260924 16:11:41.012041 2662 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:41.012106 2662 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:41.012205 2662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:41.013128 2662 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f7b39b2f24477db1f1f982089f9b62" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 39329 } }
I20260924 16:11:41.013484 2662 leader_election.cc:302] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9f7b39b2f24477db1f1f982089f9b62; no voters:
I20260924 16:11:41.013800 2662 leader_election.cc:288] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:41.013922 2667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:41.014164 2667 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [term 1 LEADER]: Becoming Leader. State: Replica: e9f7b39b2f24477db1f1f982089f9b62, State: Running, Role: LEADER
I20260924 16:11:41.014524 2667 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f7b39b2f24477db1f1f982089f9b62" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 39329 } }
I20260924 16:11:41.015108 2662 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:41.016201 2668 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9f7b39b2f24477db1f1f982089f9b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f7b39b2f24477db1f1f982089f9b62" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 39329 } } }
I20260924 16:11:41.016399 2668 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:41.016798 2676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:41.016768 2669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f7b39b2f24477db1f1f982089f9b62. Latest consensus state: current_term: 1 leader_uuid: "e9f7b39b2f24477db1f1f982089f9b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f7b39b2f24477db1f1f982089f9b62" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 39329 } } }
I20260924 16:11:41.016908 2669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:41.019899 2676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:41.025884 2676 catalog_manager.cc:1347] Generated new cluster ID: 796633e635f5431abade856d6d86655e
I20260924 16:11:41.025957 2676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:41.033146 2676 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:41.034395 2676 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:41.045785 2676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9f7b39b2f24477db1f1f982089f9b62: Generated new TSK 0
I20260924 16:11:41.046666 2676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:11:41.081079 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:39329
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:41089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:41.192996 2688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:41.193233 2688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:41.193292 2688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:41.197269 2688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:41.197343 2688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:41.197400 2688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:11:41.202397 2688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:41089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:39329
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:41.203368 2688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:41.204324 2688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:41.213889 2696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:41.214087 2694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:41.214016 2693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:41.213973 2688 server_base.cc:1034] running on GCE node
I20260924 16:11:41.214612 2688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:41.215246 2688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:41.216457 2688 hybrid_clock.cc:648] HybridClock initialized: now 1790266301216433 us; error 32 us; skew 500 ppm
Sep 24 16:11:41 dist-test-slave-6wvm krb5kdc[2578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:41.219302 2688 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:11:41.220140 2688 webserver.cc:460] Webserver started at http://127.1.253.65:35351/ using document root <none> and password file <none>
I20260924 16:11:41.220700 2688 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:41.220789 2688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:41.221020 2688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:41.223099 2688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "9f41dde183b1410b80cced163ba5ec7f"
format_stamp: "Formatted at 2026-09-24 16:11:41 on dist-test-slave-6wvm"
server_key: "7f929ac158cdbefb13e8934832d12749"
server_key_iv: "b416bbb60f837e316e73243897f72208"
server_key_version: "encryptionkey@0"
I20260924 16:11:41.223721 2688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "9f41dde183b1410b80cced163ba5ec7f"
format_stamp: "Formatted at 2026-09-24 16:11:41 on dist-test-slave-6wvm"
server_key: "7f929ac158cdbefb13e8934832d12749"
server_key_iv: "b416bbb60f837e316e73243897f72208"
server_key_version: "encryptionkey@0"
I20260924 16:11:41.227344 2688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 16:11:41.229717 2703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:41.230934 2688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:11:41.231079 2688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "9f41dde183b1410b80cced163ba5ec7f"
format_stamp: "Formatted at 2026-09-24 16:11:41 on dist-test-slave-6wvm"
server_key: "7f929ac158cdbefb13e8934832d12749"
server_key_iv: "b416bbb60f837e316e73243897f72208"
server_key_version: "encryptionkey@0"
I20260924 16:11:41.231192 2688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:41.246033 2688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:41.249348 2688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:41.249547 2688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:41.250126 2688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:41.250959 2688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:41.251005 2688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:41.251073 2688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:41.251106 2688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:41.260821 2688 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:39467
I20260924 16:11:41.260914 2820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:39467 every 8 connection(s)
I20260924 16:11:41.261787 2688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:11:41.267185 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 2688
I20260924 16:11:41.267371 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:11:41.267643 2037 external_mini_cluster.cc:1428] Setting key 55b8b0eb72e794d139c2b96218fb0d63
Sep 24 16:11:41 dist-test-slave-6wvm krb5kdc[2578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:41.277539 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.263697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:43757 (local address 127.1.253.126:39329)
0924 16:11:41.263837 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.263841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.264650 (+ 809us) server_negotiation.cc:407] Connection header received
0924 16:11:41.266125 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.266128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.266189 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.266323 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:41.267649 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.268281 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.269715 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.270136 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.270282 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.273016 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:41.273023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:41.273029 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:41.273064 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:41.274909 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.275630 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.275633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.275634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.275763 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.276261 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.276268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.276271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.276549 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:41.276672 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.277178 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.277328 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:11:41.278383 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.264067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:39329 (local address 127.1.253.65:43757)
0924 16:11:41.264505 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:41.264545 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:41.265871 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:41.266494 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:41.266501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:41.266887 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:41.267493 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.267514 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.268430 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.268434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.269577 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.269587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.270338 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.270342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.270481 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.271157 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:41.271173 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:41.272840 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:41.275036 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:41.275041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:41.275060 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:41.275483 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:41.275857 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:41.275860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:41.275862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:41.276119 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:41.276663 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:41.276668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:41.276919 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 16:11:41.277093 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 16:11:41.277366 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
I20260924 16:11:41.279729 2821 heartbeater.cc:347] Connected to a master server at 127.1.253.126:39329
I20260924 16:11:41.280006 2821 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:41.280550 2821 heartbeater.cc:511] Master 127.1.253.126:39329 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
I20260924 16:11:41.282341 2625 ts_manager.cc:194] Registered new tserver with Master: 9f41dde183b1410b80cced163ba5ec7f (127.1.253.65:39467)
I20260924 16:11:41.283494 2625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:43757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:11:41.297075 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:39329
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:41089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:41.408924 2826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:41.409198 2826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:41.409242 2826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:41.413311 2826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:41.413370 2826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:41.413460 2826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:11:41.418936 2826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:41089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:39329
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:41.419929 2826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:41.420986 2826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:41.430756 2832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:41.430965 2831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:41.430823 2826 server_base.cc:1034] running on GCE node
W20260924 16:11:41.430780 2834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:41.431558 2826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:41.432175 2826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:41.433336 2826 hybrid_clock.cc:648] HybridClock initialized: now 1790266301433324 us; error 42 us; skew 500 ppm
Sep 24 16:11:41 dist-test-slave-6wvm krb5kdc[2578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:41.436251 2826 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:11:41.437103 2826 webserver.cc:460] Webserver started at http://127.1.253.66:44913/ using document root <none> and password file <none>
I20260924 16:11:41.437668 2826 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:41.437729 2826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:41.437963 2826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:41.440017 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "04e2295887d0423a8afb1d2edb0bf5cc"
format_stamp: "Formatted at 2026-09-24 16:11:41 on dist-test-slave-6wvm"
server_key: "5b39461a0cfefa281e175b4426435ce2"
server_key_iv: "12815dcaec03d68d348027baf4e075d4"
server_key_version: "encryptionkey@0"
I20260924 16:11:41.440588 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "04e2295887d0423a8afb1d2edb0bf5cc"
format_stamp: "Formatted at 2026-09-24 16:11:41 on dist-test-slave-6wvm"
server_key: "5b39461a0cfefa281e175b4426435ce2"
server_key_iv: "12815dcaec03d68d348027baf4e075d4"
server_key_version: "encryptionkey@0"
I20260924 16:11:41.444190 2826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:11:41.446591 2841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:41.447803 2826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:11:41.447930 2826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "04e2295887d0423a8afb1d2edb0bf5cc"
format_stamp: "Formatted at 2026-09-24 16:11:41 on dist-test-slave-6wvm"
server_key: "5b39461a0cfefa281e175b4426435ce2"
server_key_iv: "12815dcaec03d68d348027baf4e075d4"
server_key_version: "encryptionkey@0"
I20260924 16:11:41.448035 2826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:41.466789 2826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:41.469811 2826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:41.470017 2826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:41.470695 2826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:41.471535 2826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:41.471613 2826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:41.471690 2826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:41.471733 2826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:41.480652 2826 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:37757
I20260924 16:11:41.480670 2958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:37757 every 8 connection(s)
I20260924 16:11:41.481632 2826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:11:41.483368 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 2826
I20260924 16:11:41.483492 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:11:41.483789 2037 external_mini_cluster.cc:1428] Setting key 71136c3026d4d002343d716e0c6976c8
Sep 24 16:11:41 dist-test-slave-6wvm krb5kdc[2578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
I20260924 16:11:41.497498 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.483687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:53733 (local address 127.1.253.126:39329)
0924 16:11:41.483840 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.483846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.484926 (+ 1080us) server_negotiation.cc:407] Connection header received
0924 16:11:41.486475 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.486481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.486548 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.486641 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:41.488260 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.488775 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.490664 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.490894 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.490978 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.493907 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:41.493910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:41.493912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:41.493937 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:41.495447 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.495962 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.495966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.495968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.496056 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.496404 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.496407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.496409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.496589 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:41.496698 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.497198 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.497319 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:11:41.498389 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.484167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:39329 (local address 127.1.253.66:53733)
0924 16:11:41.484760 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:41.484818 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:41.486193 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:41.486780 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:41.486787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:41.487335 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:41.488085 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.488109 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.488920 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.488927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.490531 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.490545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.491068 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.491075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.491331 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.492012 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:41.492041 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:41.493732 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:41.495570 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:41.495576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:41.495592 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:41.495849 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:41.496158 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:41.496161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:41.496164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:41.496304 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:41.496697 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:41.496705 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:41.496961 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 16:11:41.497156 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 16:11:41.497484 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":172,"threads_started":1}
I20260924 16:11:41.499591 2959 heartbeater.cc:347] Connected to a master server at 127.1.253.126:39329
I20260924 16:11:41.499850 2959 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:41.500386 2959 heartbeater.cc:511] Master 127.1.253.126:39329 requested a full tablet report, sending...
I20260924 16:11:41.501444 2625 ts_manager.cc:194] Registered new tserver with Master: 04e2295887d0423a8afb1d2edb0bf5cc (127.1.253.66:37757)
I20260924 16:11:41.502003 2625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:53733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:11:41.513273 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:39329
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:41089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:41.627441 2964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:41.627709 2964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:41.627755 2964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:41.632002 2964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:41.632068 2964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:41.632161 2964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:11:41.637563 2964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:41089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:39329
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:41.638736 2964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:41.639891 2964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:11:41.650002 2964 server_base.cc:1034] running on GCE node
W20260924 16:11:41.649968 2969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:41.649961 2970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:41.650161 2972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:41.650599 2964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:41.651285 2964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:41.652472 2964 hybrid_clock.cc:648] HybridClock initialized: now 1790266301652460 us; error 29 us; skew 500 ppm
Sep 24 16:11:41 dist-test-slave-6wvm krb5kdc[2578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:41.655397 2964 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:11:41.656335 2964 webserver.cc:460] Webserver started at http://127.1.253.67:36905/ using document root <none> and password file <none>
I20260924 16:11:41.656919 2964 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:41.657011 2964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:41.657264 2964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:41.659466 2964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "d117a652781146faa9740e2eb2ca36b9"
format_stamp: "Formatted at 2026-09-24 16:11:41 on dist-test-slave-6wvm"
server_key: "8aa72de06cc64e165b1a3702d2939696"
server_key_iv: "4fccdb09b1fb58bc4c96b4ebf3c302ce"
server_key_version: "encryptionkey@0"
I20260924 16:11:41.660044 2964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "d117a652781146faa9740e2eb2ca36b9"
format_stamp: "Formatted at 2026-09-24 16:11:41 on dist-test-slave-6wvm"
server_key: "8aa72de06cc64e165b1a3702d2939696"
server_key_iv: "4fccdb09b1fb58bc4c96b4ebf3c302ce"
server_key_version: "encryptionkey@0"
I20260924 16:11:41.663806 2964 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 16:11:41.666196 2979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:41.667462 2964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:11:41.667608 2964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "d117a652781146faa9740e2eb2ca36b9"
format_stamp: "Formatted at 2026-09-24 16:11:41 on dist-test-slave-6wvm"
server_key: "8aa72de06cc64e165b1a3702d2939696"
server_key_iv: "4fccdb09b1fb58bc4c96b4ebf3c302ce"
server_key_version: "encryptionkey@0"
I20260924 16:11:41.667707 2964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:41.687093 2964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:41.690162 2964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:41.690349 2964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:41.690984 2964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:41.691874 2964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:41.691952 2964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:41.692029 2964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:41.692065 2964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:41.700791 2964 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:37231
I20260924 16:11:41.700815 3094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:37231 every 8 connection(s)
I20260924 16:11:41.701682 2964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:11:41.710297 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 2964
I20260924 16:11:41.710425 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:11:41.710742 2037 external_mini_cluster.cc:1428] Setting key a08d07ca46ec643c71301d28f8b9bcbc
Sep 24 16:11:41 dist-test-slave-6wvm krb5kdc[2578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:41.716001 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.703554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:44653 (local address 127.1.253.126:39329)
0924 16:11:41.703702 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.703706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.704534 (+ 828us) server_negotiation.cc:407] Connection header received
0924 16:11:41.705916 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.705920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.705994 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.706115 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:41.707303 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.707824 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.708987 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.709212 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.709327 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.712129 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:41.712133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:41.712135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:41.712163 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:41.713681 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.714155 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.714158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.714159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.714272 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.714741 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.714744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.714745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.714928 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:41.715007 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.715613 (+ 606us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.715871 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:11:41.716838 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.703941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:39329 (local address 127.1.253.67:44653)
0924 16:11:41.704391 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:41.704433 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:41.705689 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:41.706336 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:41.706343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:41.706686 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:41.707154 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.707169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.707954 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.707958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.708856 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.708865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.709369 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.709373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.709547 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.710142 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:41.710167 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:41.711946 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:41.713788 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:41.713793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:41.713806 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:41.714066 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:41.714393 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:41.714396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:41.714397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:41.714633 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:41.715045 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:41.715051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:41.715312 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 16:11:41.715514 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 16:11:41.715895 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
I20260924 16:11:41.718288 3095 heartbeater.cc:347] Connected to a master server at 127.1.253.126:39329
I20260924 16:11:41.718564 3095 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:41.719050 3095 heartbeater.cc:511] Master 127.1.253.126:39329 requested a full tablet report, sending...
I20260924 16:11:41.720153 2625 ts_manager.cc:194] Registered new tserver with Master: d117a652781146faa9740e2eb2ca36b9 (127.1.253.67:37231)
I20260924 16:11:41.720749 2625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:44653
I20260924 16:11:41.724833 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:11:41.736864 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.729360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.1.253.126:39329)
0924 16:11:41.729525 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.729530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.729759 (+ 229us) server_negotiation.cc:407] Connection header received
0924 16:11:41.729853 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.729855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.729897 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.729991 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:41.730809 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.731334 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.732141 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.732343 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.732418 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.733712 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:41.733715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:41.733717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:41.733740 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:41.735290 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.735752 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.735755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.735756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.735826 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.736080 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.736082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.736083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.736278 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:41.736380 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.736573 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.736662 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:11:41.742411 2625 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:11:41.745616 2625 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:11:41.760735 3107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.754699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:37757 (local address 127.0.0.1:59970)
0924 16:11:41.755509 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:41.755521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:41.755594 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:41.756311 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:41.756315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:41.756351 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:41.756584 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.756592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.758037 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.758041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.759272 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.759283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.760130 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.760134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.760257 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.760297 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:11:41.760437 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 16:11:41.760557 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":81,"spinlock_wait_cycles":6144,"thread_start_us":81,"threads_started":1}
I20260924 16:11:41.760941 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.754737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.1.253.67:37231)
0924 16:11:41.755785 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.755797 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.755820 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:11:41.755920 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.755926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.756143 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.756704 (+ 561us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:41.756865 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.757660 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.759440 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.760301 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.760435 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.760564 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.760671 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.760767 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":171,"threads_started":1}
I20260924 16:11:41.760879 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.754592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:37231 (local address 127.0.0.1:60312)
0924 16:11:41.755313 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:41.755364 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:41.755494 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:41.756305 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:41.756311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:41.756349 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:41.756564 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.756572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.757794 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.757799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.759280 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.759287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.760443 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.760447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.760534 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.760553 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 16:11:41.760642 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 16:11:41.760698 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":55,"spinlock_wait_cycles":11392,"thread_start_us":87,"threads_started":1}
I20260924 16:11:41.761610 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.755417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.1.253.66:37757)
0924 16:11:41.755825 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.755837 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.755859 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:11:41.755936 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.755942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.756147 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.756460 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:41.756706 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.757904 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.759408 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.759986 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.760118 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.760283 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.761314 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.761436 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
I20260924 16:11:41.762131 3106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.754977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:39467 (local address 127.0.0.1:50032)
0924 16:11:41.755954 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:41.755967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:41.756044 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:41.756727 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:41.756730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:41.756746 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:41.756836 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.756842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.758213 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.758217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.759310 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.759319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.760648 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.760651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.760735 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.760757 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:11:41.760829 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 16:11:41.760909 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":26,"thread_start_us":79,"threads_started":1}
I20260924 16:11:41.762477 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.755415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.1.253.65:39467)
0924 16:11:41.755827 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.755841 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.756128 (+ 287us) server_negotiation.cc:407] Connection header received
0924 16:11:41.756211 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.756219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.756401 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.756555 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:41.756976 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.758093 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.759453 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.760301 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.760424 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.760564 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.761161 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.761285 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":106,"threads_started":1}
I20260924 16:11:41.764324 2891 tablet_service.cc:1467] Processing CreateTablet for tablet 257d4f7b29e84a8e9e3e8f04fdced9c1 (DEFAULT_TABLE table=test-table [id=ddffc8b56ebb43d3ad679a47129690d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:41.764451 3028 tablet_service.cc:1467] Processing CreateTablet for tablet 257d4f7b29e84a8e9e3e8f04fdced9c1 (DEFAULT_TABLE table=test-table [id=ddffc8b56ebb43d3ad679a47129690d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:41.765054 2752 tablet_service.cc:1467] Processing CreateTablet for tablet 257d4f7b29e84a8e9e3e8f04fdced9c1 (DEFAULT_TABLE table=test-table [id=ddffc8b56ebb43d3ad679a47129690d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:41.765429 2891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 257d4f7b29e84a8e9e3e8f04fdced9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:41.765484 3028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 257d4f7b29e84a8e9e3e8f04fdced9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:41.765977 2752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 257d4f7b29e84a8e9e3e8f04fdced9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:41.771871 3113 tablet_bootstrap.cc:492] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc: Bootstrap starting.
I20260924 16:11:41.773095 3115 tablet_bootstrap.cc:492] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f: Bootstrap starting.
I20260924 16:11:41.773636 3114 tablet_bootstrap.cc:492] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9: Bootstrap starting.
I20260924 16:11:41.775326 3115 tablet_bootstrap.cc:654] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:41.776430 3114 tablet_bootstrap.cc:654] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:41.776710 3113 tablet_bootstrap.cc:654] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:41.777089 3115 log.cc:824] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:41.777344 3114 log.cc:824] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:41.777439 3113 log.cc:824] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:41.779224 3115 tablet_bootstrap.cc:492] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f: No bootstrap required, opened a new log
I20260924 16:11:41.779328 3113 tablet_bootstrap.cc:492] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc: No bootstrap required, opened a new log
I20260924 16:11:41.779328 3114 tablet_bootstrap.cc:492] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9: No bootstrap required, opened a new log
I20260924 16:11:41.779443 3115 ts_tablet_manager.cc:1397] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 16:11:41.779510 3113 ts_tablet_manager.cc:1397] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 16:11:41.779510 3114 ts_tablet_manager.cc:1397] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:11:41.782224 3115 raft_consensus.cc:348] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d117a652781146faa9740e2eb2ca36b9" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37231 } } peers { permanent_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37757 } } peers { permanent_uuid: "9f41dde183b1410b80cced163ba5ec7f" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 39467 } }
I20260924 16:11:41.782580 3115 raft_consensus.cc:374] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:41.782666 3115 raft_consensus.cc:729] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f41dde183b1410b80cced163ba5ec7f, State: Initialized, Role: FOLLOWER
I20260924 16:11:41.783198 3115 consensus_queue.cc:260] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d117a652781146faa9740e2eb2ca36b9" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37231 } } peers { permanent_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37757 } } peers { permanent_uuid: "9f41dde183b1410b80cced163ba5ec7f" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 39467 } }
I20260924 16:11:41.783973 2821 heartbeater.cc:503] Master 127.1.253.126:39329 was elected leader, sending a full tablet report...
I20260924 16:11:41.784245 3115 ts_tablet_manager.cc:1428] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260924 16:11:41.784235 3113 raft_consensus.cc:348] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d117a652781146faa9740e2eb2ca36b9" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37231 } } peers { permanent_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37757 } } peers { permanent_uuid: "9f41dde183b1410b80cced163ba5ec7f" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 39467 } }
I20260924 16:11:41.784502 3113 raft_consensus.cc:374] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:41.784559 3113 raft_consensus.cc:729] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e2295887d0423a8afb1d2edb0bf5cc, State: Initialized, Role: FOLLOWER
I20260924 16:11:41.785022 3113 consensus_queue.cc:260] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d117a652781146faa9740e2eb2ca36b9" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37231 } } peers { permanent_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37757 } } peers { permanent_uuid: "9f41dde183b1410b80cced163ba5ec7f" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 39467 } }
I20260924 16:11:41.786167 3113 ts_tablet_manager.cc:1428] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 16:11:41.786167 3114 raft_consensus.cc:348] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d117a652781146faa9740e2eb2ca36b9" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37231 } } peers { permanent_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37757 } } peers { permanent_uuid: "9f41dde183b1410b80cced163ba5ec7f" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 39467 } }
I20260924 16:11:41.786577 3114 raft_consensus.cc:374] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:41.786585 2959 heartbeater.cc:503] Master 127.1.253.126:39329 was elected leader, sending a full tablet report...
I20260924 16:11:41.786669 3114 raft_consensus.cc:729] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d117a652781146faa9740e2eb2ca36b9, State: Initialized, Role: FOLLOWER
I20260924 16:11:41.787098 3114 consensus_queue.cc:260] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d117a652781146faa9740e2eb2ca36b9" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37231 } } peers { permanent_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37757 } } peers { permanent_uuid: "9f41dde183b1410b80cced163ba5ec7f" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 39467 } }
I20260924 16:11:41.787894 3095 heartbeater.cc:503] Master 127.1.253.126:39329 was elected leader, sending a full tablet report...
I20260924 16:11:41.788280 3114 ts_tablet_manager.cc:1428] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260924 16:11:41.825124 3120 raft_consensus.cc:482] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:41.825376 3120 raft_consensus.cc:504] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d117a652781146faa9740e2eb2ca36b9" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37231 } } peers { permanent_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37757 } } peers { permanent_uuid: "9f41dde183b1410b80cced163ba5ec7f" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 39467 } }
I20260924 16:11:41.826721 3120 leader_election.cc:288] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d117a652781146faa9740e2eb2ca36b9 (127.1.253.67:37231), 9f41dde183b1410b80cced163ba5ec7f (127.1.253.65:39467)
I20260924 16:11:41.831022 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.826971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:37231 (local address 127.1.253.66:60023)
0924 16:11:41.827141 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:41.827157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:41.827269 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:41.827716 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:41.827718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:41.827736 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:41.827952 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.827960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.828811 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.828814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.829751 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.829761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.830552 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.830556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.830662 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.830693 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:11:41.830767 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 16:11:41.830836 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":41,"spinlock_wait_cycles":14592}
I20260924 16:11:41.831038 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.827197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:60023 (local address 127.1.253.67:37231)
0924 16:11:41.827457 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.827461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.827479 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:11:41.827538 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.827542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.827609 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.827695 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:41.828106 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.828700 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.829872 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.830452 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.830559 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.830613 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.830839 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.830909 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260924 16:11:41.831174 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.827206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:60263 (local address 127.1.253.65:39467)
0924 16:11:41.827379 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.827385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.827783 (+ 398us) server_negotiation.cc:407] Connection header received
0924 16:11:41.827939 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.827943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.827998 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.828095 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:41.828357 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.828981 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.830079 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.830602 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.830721 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.830768 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.830967 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.831040 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:11:41.831552 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.826971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:39467 (local address 127.1.253.66:60263)
0924 16:11:41.827683 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:41.827697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:41.827803 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:41.828107 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:41.828110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:41.828128 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:41.828232 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.828238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.829090 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.829093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.829960 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:41.829970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.830725 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.830731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:41.830831 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.830850 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 16:11:41.831308 (+ 458us) client_negotiation.cc:240] Negotiation successful
0924 16:11:41.831426 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":151,"thread_start_us":105,"threads_started":1}
I20260924 16:11:41.832824 2773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257d4f7b29e84a8e9e3e8f04fdced9c1" candidate_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f41dde183b1410b80cced163ba5ec7f" is_pre_election: true
I20260924 16:11:41.832866 3048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257d4f7b29e84a8e9e3e8f04fdced9c1" candidate_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d117a652781146faa9740e2eb2ca36b9" is_pre_election: true
I20260924 16:11:41.833169 2773 raft_consensus.cc:2454] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04e2295887d0423a8afb1d2edb0bf5cc in term 0.
I20260924 16:11:41.833295 3048 raft_consensus.cc:2454] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04e2295887d0423a8afb1d2edb0bf5cc in term 0.
I20260924 16:11:41.833669 2842 leader_election.cc:302] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e2295887d0423a8afb1d2edb0bf5cc, 9f41dde183b1410b80cced163ba5ec7f; no voters:
I20260924 16:11:41.833971 3120 raft_consensus.cc:2781] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:41.834048 3120 raft_consensus.cc:482] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:41.834077 3120 raft_consensus.cc:3037] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:41.835325 3120 raft_consensus.cc:504] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d117a652781146faa9740e2eb2ca36b9" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37231 } } peers { permanent_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37757 } } peers { permanent_uuid: "9f41dde183b1410b80cced163ba5ec7f" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 39467 } }
I20260924 16:11:41.835685 3120 leader_election.cc:288] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [CANDIDATE]: Term 1 election: Requested vote from peers d117a652781146faa9740e2eb2ca36b9 (127.1.253.67:37231), 9f41dde183b1410b80cced163ba5ec7f (127.1.253.65:39467)
I20260924 16:11:41.836072 3048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257d4f7b29e84a8e9e3e8f04fdced9c1" candidate_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d117a652781146faa9740e2eb2ca36b9"
I20260924 16:11:41.836112 2773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257d4f7b29e84a8e9e3e8f04fdced9c1" candidate_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f41dde183b1410b80cced163ba5ec7f"
I20260924 16:11:41.836187 3048 raft_consensus.cc:3037] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:41.836252 2773 raft_consensus.cc:3037] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:41.837292 3048 raft_consensus.cc:2454] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P d117a652781146faa9740e2eb2ca36b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04e2295887d0423a8afb1d2edb0bf5cc in term 1.
I20260924 16:11:41.837512 2773 raft_consensus.cc:2454] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 9f41dde183b1410b80cced163ba5ec7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04e2295887d0423a8afb1d2edb0bf5cc in term 1.
I20260924 16:11:41.837612 2845 leader_election.cc:302] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e2295887d0423a8afb1d2edb0bf5cc, d117a652781146faa9740e2eb2ca36b9; no voters:
I20260924 16:11:41.837767 3120 raft_consensus.cc:2781] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:41.838003 3120 raft_consensus.cc:686] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [term 1 LEADER]: Becoming Leader. State: Replica: 04e2295887d0423a8afb1d2edb0bf5cc, State: Running, Role: LEADER
I20260924 16:11:41.838404 3120 consensus_queue.cc:237] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d117a652781146faa9740e2eb2ca36b9" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37231 } } peers { permanent_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37757 } } peers { permanent_uuid: "9f41dde183b1410b80cced163ba5ec7f" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 39467 } }
I20260924 16:11:41.842072 2624 catalog_manager.cc:5495] T 257d4f7b29e84a8e9e3e8f04fdced9c1 P 04e2295887d0423a8afb1d2edb0bf5cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 04e2295887d0423a8afb1d2edb0bf5cc (127.1.253.66). New cstate: current_term: 1 leader_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d117a652781146faa9740e2eb2ca36b9" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04e2295887d0423a8afb1d2edb0bf5cc" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f41dde183b1410b80cced163ba5ec7f" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 39467 } health_report { overall_health: UNKNOWN } } }
Sep 24 16:11:41 dist-test-slave-6wvm krb5kdc[2578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.65@KRBTEST.COM
I20260924 16:11:41.861640 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.852104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.1.253.65:39467)
0924 16:11:41.852318 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.852323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.852527 (+ 204us) server_negotiation.cc:407] Connection header received
0924 16:11:41.852704 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.852712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.852763 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.852857 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:41.853643 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.854169 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.855079 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.855305 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.855422 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.858063 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:41.858076 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:41.858092 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:41.858119 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:41.859780 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.860283 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.860290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.860291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.860379 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.860681 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.860685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.860687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.860949 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:41.861184 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.861341 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.861470 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
Sep 24 16:11:41 dist-test-slave-6wvm krb5kdc[2578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:11:41 dist-test-slave-6wvm krb5kdc[2578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.253.65@KRBTEST.COM
I20260924 16:11:41.890015 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.881427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.1.253.65:39467)
0924 16:11:41.881605 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.881611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.881720 (+ 109us) server_negotiation.cc:407] Connection header received
0924 16:11:41.881906 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.881910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.881961 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.882052 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:41.882814 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.883337 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.884187 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.884420 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.884533 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.887023 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:41.887026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:41.887028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:41.887054 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:41.888322 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.888743 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.888746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.888748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.888827 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.889104 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.889108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.889110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.889279 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:41.889432 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.889715 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.889852 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 16:11:41 dist-test-slave-6wvm krb5kdc[2578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:11:41 dist-test-slave-6wvm krb5kdc[2578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.65@KRBTEST.COM
I20260924 16:11:41.916668 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:41.908025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.1.253.65:39467)
0924 16:11:41.908197 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:41.908202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:41.908327 (+ 125us) server_negotiation.cc:407] Connection header received
0924 16:11:41.908546 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:41.908551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:41.908606 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:41.908701 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:41.909428 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.909948 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.910940 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:41.911165 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:41.911284 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:41.913731 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:41.913735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:41.913737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:41.913759 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:41.915017 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.915442 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.915446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.915448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.915528 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:41.915793 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:41.915797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:41.915799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:41.915962 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:41.916105 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:41.916381 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 16:11:41.916504 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 16:11:41.918327 2731 tablet_service.cc:2908] Rejecting scan request for tablet 257d4f7b29e84a8e9e3e8f04fdced9c1: Uninitialized: clean time has not yet been initialized
I20260924 16:11:41.921003 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 2688
I20260924 16:11:41.927814 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 2826
I20260924 16:11:41.933925 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 2964
I20260924 16:11:41.939857 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 2592
2026-09-24T16:11:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2297 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:42 dist-test-slave-6wvm krb5kdc[3137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:42 dist-test-slave-6wvm krb5kdc[3137](info): set up 2 sockets
Sep 24 16:11:42 dist-test-slave-6wvm krb5kdc[3137](info): commencing operation
krb5kdc: starting...
W20260924 16:11:42.993456 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:11:43 dist-test-slave-6wvm krb5kdc[3137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:11:43.117194 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:36927
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:43.230958 3151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:43.231269 3151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:43.231361 3151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:43.235481 3151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:43.235565 3151 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:11:43.235594 3151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:43.235639 3151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:43.235682 3151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:43.235718 3151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:43.241065 3151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:36927
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:43.242199 3151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:43.243345 3151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:43.252064 3156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:43.252084 3151 server_base.cc:1034] running on GCE node
W20260924 16:11:43.252060 3157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:43.252060 3159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:43.253088 3151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:43.254333 3151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:43.255522 3151 hybrid_clock.cc:648] HybridClock initialized: now 1790266303255515 us; error 57 us; skew 500 ppm
Sep 24 16:11:43 dist-test-slave-6wvm krb5kdc[3137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:43.265146 3151 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:11:43.266068 3151 webserver.cc:460] Webserver started at http://127.1.253.126:35965/ using document root <none> and password file <none>
I20260924 16:11:43.266661 3151 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:43.266724 3151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:43.266899 3151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:43.268909 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "9a06e995408b4af0ae34ce41e0d88b19"
format_stamp: "Formatted at 2026-09-24 16:11:43 on dist-test-slave-6wvm"
server_key: "95ec754b2642a7208c3027cd9b340e0c"
server_key_iv: "4f4f72c596f6eea63c192543fde2f95d"
server_key_version: "encryptionkey@0"
I20260924 16:11:43.269423 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "9a06e995408b4af0ae34ce41e0d88b19"
format_stamp: "Formatted at 2026-09-24 16:11:43 on dist-test-slave-6wvm"
server_key: "95ec754b2642a7208c3027cd9b340e0c"
server_key_iv: "4f4f72c596f6eea63c192543fde2f95d"
server_key_version: "encryptionkey@0"
I20260924 16:11:43.273233 3151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 16:11:43.275837 3166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:43.277171 3151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:11:43.277331 3151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "9a06e995408b4af0ae34ce41e0d88b19"
format_stamp: "Formatted at 2026-09-24 16:11:43 on dist-test-slave-6wvm"
server_key: "95ec754b2642a7208c3027cd9b340e0c"
server_key_iv: "4f4f72c596f6eea63c192543fde2f95d"
server_key_version: "encryptionkey@0"
I20260924 16:11:43.277451 3151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:43.294811 3151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:43.297750 3151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:43.297966 3151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:43.306107 3151 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:36927
I20260924 16:11:43.306121 3229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:36927 every 8 connection(s)
I20260924 16:11:43.307327 3151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:43.310526 3230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:43.313567 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 3151
I20260924 16:11:43.313714 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:11:43.314029 2037 external_mini_cluster.cc:1428] Setting key bfc65f610c688d0aa61a0de7b11e2426
I20260924 16:11:43.317939 3230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19: Bootstrap starting.
I20260924 16:11:43.320736 3230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:43.321619 3230 log.cc:824] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:43.323609 3230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19: No bootstrap required, opened a new log
Sep 24 16:11:43 dist-test-slave-6wvm krb5kdc[3137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:43.327395 3230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a06e995408b4af0ae34ce41e0d88b19" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 36927 } }
I20260924 16:11:43.327649 3230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:43.327744 3230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a06e995408b4af0ae34ce41e0d88b19, State: Initialized, Role: FOLLOWER
I20260924 16:11:43.328346 3230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a06e995408b4af0ae34ce41e0d88b19" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 36927 } }
I20260924 16:11:43.328503 3230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:43.328599 3230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:43.328707 3230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:43.329622 3230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a06e995408b4af0ae34ce41e0d88b19" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 36927 } }
I20260924 16:11:43.329905 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:43.316449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.1.253.126:36927)
0924 16:11:43.316868 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:43.316895 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:43.316943 (+ 48us) server_negotiation.cc:407] Connection header received
0924 16:11:43.317559 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:43.317577 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:43.317927 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:43.318306 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:43.319224 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.320240 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.321401 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.321950 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.322127 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:43.324696 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:43.324703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:43.324726 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:43.324754 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:43.326634 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.327100 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.327108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.327120 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.327247 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.327548 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.327551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.327553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.328163 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:43.328404 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:43.328730 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 16:11:43.329078 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":134,"threads_started":1}
I20260924 16:11:43.330005 3230 leader_election.cc:302] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a06e995408b4af0ae34ce41e0d88b19; no voters:
I20260924 16:11:43.330399 3230 leader_election.cc:288] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:43.330509 3236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:43.330729 3236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [term 1 LEADER]: Becoming Leader. State: Replica: 9a06e995408b4af0ae34ce41e0d88b19, State: Running, Role: LEADER
I20260924 16:11:43.331107 3236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a06e995408b4af0ae34ce41e0d88b19" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 36927 } }
I20260924 16:11:43.331751 3230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:43.332921 3238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a06e995408b4af0ae34ce41e0d88b19. Latest consensus state: current_term: 1 leader_uuid: "9a06e995408b4af0ae34ce41e0d88b19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a06e995408b4af0ae34ce41e0d88b19" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 36927 } } }
I20260924 16:11:43.332921 3237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a06e995408b4af0ae34ce41e0d88b19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a06e995408b4af0ae34ce41e0d88b19" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 36927 } } }
I20260924 16:11:43.333077 3237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:43.333077 3238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19 [sys.catalog]: This master's current role is: LEADER
W20260924 16:11:43.336706 3252 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:11:43.336817 3252 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:11:43.336908 3246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:43.337872 3246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:43.346879 3246 catalog_manager.cc:1347] Generated new cluster ID: a1b60c99c3184cb9a0e6f585f34a48ba
I20260924 16:11:43.346952 3246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:43.354715 3246 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:43.355973 3246 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:43.370216 3246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19: Generated new TSK 0
I20260924 16:11:43.371085 3246 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:11:43.406988 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:36927
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:43.521912 3258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:43.522192 3258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:43.522300 3258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:43.526338 3258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:43.526391 3258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:43.526427 3258 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:11:43.526459 3258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:11:43.531967 3258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.253.126:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:43.533200 3258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:43.534309 3258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:43.543834 3263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:43.543922 3264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:43.543829 3266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:43.544049 3258 server_base.cc:1034] running on GCE node
I20260924 16:11:43.544519 3258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:43.545171 3258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:43.546322 3258 hybrid_clock.cc:648] HybridClock initialized: now 1790266303546303 us; error 35 us; skew 500 ppm
Sep 24 16:11:43 dist-test-slave-6wvm krb5kdc[3137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:43.549129 3258 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:11:43.549933 3258 webserver.cc:460] Webserver started at http://127.1.253.65:35023/ using document root <none> and password file <none>
I20260924 16:11:43.550551 3258 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:43.550614 3258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:43.550838 3258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:43.552867 3258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "271109896a3c4443bf1e4bcca45699fc"
format_stamp: "Formatted at 2026-09-24 16:11:43 on dist-test-slave-6wvm"
server_key: "f3463b8e6470e3e74e383eebde3b7d60"
server_key_iv: "0aaa85a830fd3c89fd24757889a8809c"
server_key_version: "encryptionkey@0"
I20260924 16:11:43.553450 3258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "271109896a3c4443bf1e4bcca45699fc"
format_stamp: "Formatted at 2026-09-24 16:11:43 on dist-test-slave-6wvm"
server_key: "f3463b8e6470e3e74e383eebde3b7d60"
server_key_iv: "0aaa85a830fd3c89fd24757889a8809c"
server_key_version: "encryptionkey@0"
I20260924 16:11:43.557184 3258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 16:11:43.560539 3273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:43.561707 3258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:11:43.561867 3258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "271109896a3c4443bf1e4bcca45699fc"
format_stamp: "Formatted at 2026-09-24 16:11:43 on dist-test-slave-6wvm"
server_key: "f3463b8e6470e3e74e383eebde3b7d60"
server_key_iv: "0aaa85a830fd3c89fd24757889a8809c"
server_key_version: "encryptionkey@0"
I20260924 16:11:43.561980 3258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:43.576239 3258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:43.579404 3258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:43.579612 3258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:43.581204 3258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:43.581275 3258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:43.581348 3258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:43.581382 3258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:11:43 dist-test-slave-6wvm krb5kdc[3137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:43.592303 3258 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:34819
I20260924 16:11:43.592597 3390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:34819 every 8 connection(s)
I20260924 16:11:43.593451 3258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:11:43.593833 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 3258
I20260924 16:11:43.593958 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:11:43.594302 2037 external_mini_cluster.cc:1428] Setting key d96c11a44e5ac9cd641214c1f411574a
I20260924 16:11:43.596469 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:43.582628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:36927 (local address 127.1.253.65:51909)
0924 16:11:43.583110 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:43.583152 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:43.584143 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:43.584937 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:43.584949 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:43.585429 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:43.586163 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:43.586187 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.587338 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.587341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:43.588336 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:43.588347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.588893 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.588897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:43.589083 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:43.589659 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:43.589676 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:43.591307 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:43.593343 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:43.593347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:43.593358 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:43.593695 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:43.594058 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:43.594064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:43.594068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:43.594265 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:43.594771 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:43.594777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:43.595080 (+ 303us) client_negotiation.cc:769] Sending connection context
0924 16:11:43.595273 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 16:11:43.595602 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":140,"threads_started":1}
I20260924 16:11:43.597270 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:43.581994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:51909 (local address 127.1.253.126:36927)
0924 16:11:43.582127 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:43.582130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:43.583251 (+ 1121us) server_negotiation.cc:407] Connection header received
0924 16:11:43.584404 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:43.584411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:43.584483 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:43.584592 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:43.586366 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.587218 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.588471 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.588723 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.588837 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:43.591489 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:43.591493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:43.591496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:43.591527 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:43.593231 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.593832 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.593837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.593842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.593944 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.594379 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.594384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.594386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.594628 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:43.594740 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:43.597057 (+ 2317us) server_negotiation.cc:299] Negotiation successful
0924 16:11:43.597162 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:11:43.598551 3391 heartbeater.cc:347] Connected to a master server at 127.1.253.126:36927
I20260924 16:11:43.598722 3391 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:43.599344 3391 heartbeater.cc:511] Master 127.1.253.126:36927 requested a full tablet report, sending...
I20260924 16:11:43.601167 3182 ts_manager.cc:194] Registered new tserver with Master: 271109896a3c4443bf1e4bcca45699fc (127.1.253.65:34819)
I20260924 16:11:43.602984 3182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:51909
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
I20260924 16:11:43.608983 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:43.601007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:33865 (local address 127.1.253.126:36927)
0924 16:11:43.601131 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:43.601137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:43.601320 (+ 183us) server_negotiation.cc:407] Connection header received
0924 16:11:43.601488 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:43.601491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:43.601532 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:43.601622 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:43.602058 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.602761 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.604022 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.604401 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.604521 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:43.605819 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:43.605822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:43.605825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:43.605849 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:43.607277 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.607731 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.607734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.607735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.607807 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.608114 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.608120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.608125 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.608371 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:43.608467 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:43.608776 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 16:11:43.608867 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":47}
I20260924 16:11:43.609198 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:43.600923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:36927 (local address 127.1.253.65:33865)
0924 16:11:43.601241 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:43.601256 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:43.601388 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:43.601640 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:43.601643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:43.601841 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:43.601943 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:43.601948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.602900 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.602903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:43.603633 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:43.603645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.604547 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.604551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:43.604647 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:43.605244 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:43.605258 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:43.605684 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:43.607450 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:43.607453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:43.607454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:43.607645 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:43.607910 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:43.607916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:43.607921 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:43.608011 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:43.608488 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:43.608492 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:43.608594 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 16:11:43.608716 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 16:11:43.608888 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:11:43.625679 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:36927
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:43.742578 3400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:43.742870 3400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:43.742949 3400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:43.746946 3400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:43.747037 3400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:43.747108 3400 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:11:43.747193 3400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:11:43.752638 3400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.253.126:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:43.753791 3400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:43.754894 3400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:43.764662 3406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:43.764647 3405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:43.764652 3408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:43.765079 3400 server_base.cc:1034] running on GCE node
I20260924 16:11:43.765507 3400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:43.766170 3400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:43.767362 3400 hybrid_clock.cc:648] HybridClock initialized: now 1790266303767338 us; error 41 us; skew 500 ppm
Sep 24 16:11:43 dist-test-slave-6wvm krb5kdc[3137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:43.770102 3400 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:11:43.771049 3400 webserver.cc:460] Webserver started at http://127.1.253.66:32843/ using document root <none> and password file <none>
I20260924 16:11:43.771611 3400 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:43.771703 3400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:43.771940 3400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:43.773947 3400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "159a61e457e8439c9c2c96d318633c17"
format_stamp: "Formatted at 2026-09-24 16:11:43 on dist-test-slave-6wvm"
server_key: "d8ccf6e22d70a813da39ccf2f6a0ad78"
server_key_iv: "9a29fd77f22c650f93b289938c12a1aa"
server_key_version: "encryptionkey@0"
I20260924 16:11:43.774552 3400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "159a61e457e8439c9c2c96d318633c17"
format_stamp: "Formatted at 2026-09-24 16:11:43 on dist-test-slave-6wvm"
server_key: "d8ccf6e22d70a813da39ccf2f6a0ad78"
server_key_iv: "9a29fd77f22c650f93b289938c12a1aa"
server_key_version: "encryptionkey@0"
I20260924 16:11:43.778385 3400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:11:43.780921 3415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:43.782300 3400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:11:43.782434 3400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "159a61e457e8439c9c2c96d318633c17"
format_stamp: "Formatted at 2026-09-24 16:11:43 on dist-test-slave-6wvm"
server_key: "d8ccf6e22d70a813da39ccf2f6a0ad78"
server_key_iv: "9a29fd77f22c650f93b289938c12a1aa"
server_key_version: "encryptionkey@0"
I20260924 16:11:43.782558 3400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:43.808240 3400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:43.811412 3400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:43.811636 3400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:43.813228 3400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:43.813301 3400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:43.813378 3400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:43.813422 3400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:11:43 dist-test-slave-6wvm krb5kdc[3137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:43.824569 3400 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:42917
I20260924 16:11:43.824604 3530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:42917 every 8 connection(s)
I20260924 16:11:43.825630 3400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:11:43.827754 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:43.814027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:50427 (local address 127.1.253.126:36927)
0924 16:11:43.814183 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:43.814188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:43.815224 (+ 1036us) server_negotiation.cc:407] Connection header received
0924 16:11:43.816613 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:43.816619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:43.816686 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:43.816785 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:43.818213 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.818803 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.820126 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.820362 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.820502 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:43.823796 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:43.823799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:43.823801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:43.823826 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:43.825512 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.826136 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.826141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.826143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.826272 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.826708 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.826711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.826712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.826908 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:43.826993 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:43.827525 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 16:11:43.827633 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:11:43.828627 3424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:43.814505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:36927 (local address 127.1.253.66:50427)
0924 16:11:43.815036 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:43.815100 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:43.816278 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:43.816995 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:43.817003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:43.817446 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:43.818063 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:43.818081 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.818932 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.818936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:43.819985 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:43.819996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.820559 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.820563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:43.820759 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:43.821393 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:43.821419 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:43.823582 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:43.825633 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:43.825641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:43.825658 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:43.826001 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:43.826415 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:43.826418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:43.826421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:43.826594 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:43.827036 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:43.827042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:43.827282 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 16:11:43.827480 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 16:11:43.827733 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260924 16:11:43.829849 3531 heartbeater.cc:347] Connected to a master server at 127.1.253.126:36927
I20260924 16:11:43.829985 3531 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:43.830577 3531 heartbeater.cc:511] Master 127.1.253.126:36927 requested a full tablet report, sending...
I20260924 16:11:43.831717 3183 ts_manager.cc:194] Registered new tserver with Master: 159a61e457e8439c9c2c96d318633c17 (127.1.253.66:42917)
I20260924 16:11:43.832540 3183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:50427
I20260924 16:11:43.832727 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 3400
I20260924 16:11:43.832896 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:11:43.833222 2037 external_mini_cluster.cc:1428] Setting key f2e6dcc8075a8239f013e6d8dc8a8752
I20260924 16:11:43.840310 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:43.832357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:58493 (local address 127.1.253.126:36927)
0924 16:11:43.832459 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:43.832463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:43.832925 (+ 462us) server_negotiation.cc:407] Connection header received
0924 16:11:43.833051 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:43.833055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:43.833109 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:43.833192 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:43.833719 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.834466 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.835699 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.835921 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.836006 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:43.837207 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:43.837211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:43.837214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:43.837237 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:43.838645 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.839072 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.839075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.839077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.839157 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:43.839450 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:43.839454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:43.839456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:43.839669 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:43.839780 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:43.840039 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 16:11:43.840170 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 16:11:43.840322 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:43.832267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:36927 (local address 127.1.253.66:58493)
0924 16:11:43.832776 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:43.832808 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:43.832928 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:43.833227 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:43.833231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:43.833452 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:43.833570 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:43.833577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.834650 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.834654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:43.835556 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:43.835581 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:43.836039 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:43.836046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:43.836179 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:43.836764 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:43.836775 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:43.837069 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:43.838755 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:43.838759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:43.838761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:43.838956 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:43.839253 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:43.839259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:43.839264 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:43.839368 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:43.839771 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:43.839774 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:43.839874 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 16:11:43.839984 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 16:11:43.840139 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:11:43.883435 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:36927
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:43.996884 3540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:43.997150 3540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:43.997195 3540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:44.001935 3540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:44.001996 3540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:44.002070 3540 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:11:44.002144 3540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:11:44.007550 3540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.253.126:36927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:44.008672 3540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:44.009754 3540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:44.019810 3548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:44.019850 3545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:44.019870 3546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:44.020222 3540 server_base.cc:1034] running on GCE node
I20260924 16:11:44.020634 3540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:44.021293 3540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:44.022451 3540 hybrid_clock.cc:648] HybridClock initialized: now 1790266304022438 us; error 27 us; skew 500 ppm
Sep 24 16:11:44 dist-test-slave-6wvm krb5kdc[3137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266304, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:44.025319 3540 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:11:44.026301 3540 webserver.cc:460] Webserver started at http://127.1.253.67:36635/ using document root <none> and password file <none>
I20260924 16:11:44.026906 3540 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:44.026969 3540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:44.027221 3540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:44.029482 3540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "4820bd99d2c74ebea2b72e51cf262902"
format_stamp: "Formatted at 2026-09-24 16:11:44 on dist-test-slave-6wvm"
server_key: "bc0347a403d4ba236064512565180f3c"
server_key_iv: "1170ad1e4d5409e48b9e6c0f7d23b667"
server_key_version: "encryptionkey@0"
I20260924 16:11:44.030179 3540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "4820bd99d2c74ebea2b72e51cf262902"
format_stamp: "Formatted at 2026-09-24 16:11:44 on dist-test-slave-6wvm"
server_key: "bc0347a403d4ba236064512565180f3c"
server_key_iv: "1170ad1e4d5409e48b9e6c0f7d23b667"
server_key_version: "encryptionkey@0"
I20260924 16:11:44.034680 3540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:11:44.037176 3556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:44.038436 3540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:11:44.038576 3540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "4820bd99d2c74ebea2b72e51cf262902"
format_stamp: "Formatted at 2026-09-24 16:11:44 on dist-test-slave-6wvm"
server_key: "bc0347a403d4ba236064512565180f3c"
server_key_iv: "1170ad1e4d5409e48b9e6c0f7d23b667"
server_key_version: "encryptionkey@0"
I20260924 16:11:44.038678 3540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:44.067557 3540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:44.070686 3540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:44.070888 3540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:44.072381 3540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:44.072425 3540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:44.072463 3540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:44.072520 3540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:11:44 dist-test-slave-6wvm krb5kdc[3137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266304, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:44.083817 3540 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:45585
I20260924 16:11:44.084090 3673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:45585 every 8 connection(s)
I20260924 16:11:44.085011 3540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:11:44.088240 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.073343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:57471 (local address 127.1.253.126:36927)
0924 16:11:44.073486 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.073489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.074718 (+ 1229us) server_negotiation.cc:407] Connection header received
0924 16:11:44.075899 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.075906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.075990 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.076054 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:44.077899 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.078868 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.080151 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.080455 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.080588 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.083779 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:44.083783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:44.083786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:44.083815 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:44.085713 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:44.086365 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:44.086372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:44.086376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:44.086464 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:44.086852 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:44.086862 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:44.086864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:44.087126 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:44.087224 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.087961 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.088081 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:11:44.089217 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.073906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:36927 (local address 127.1.253.67:57471)
0924 16:11:44.074531 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.074590 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.075603 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.076547 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.076554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.077004 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:44.077702 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.077725 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.079014 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.079019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.080000 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.080010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.080658 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.080663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.080872 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.081498 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:44.081527 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:44.083558 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:44.085856 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:44.085864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:44.085880 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:44.086227 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:44.086564 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:44.086568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:44.086570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:44.086730 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:44.087242 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:44.087250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:44.087597 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.087846 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.088161 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":153,"threads_started":1}
I20260924 16:11:44.090691 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 3540
I20260924 16:11:44.090806 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:11:44.090818 3674 heartbeater.cc:347] Connected to a master server at 127.1.253.126:36927
I20260924 16:11:44.090971 3674 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:44.091164 2037 external_mini_cluster.cc:1428] Setting key 96296d8e29fe90094a4e7b0f4f322516
I20260924 16:11:44.091570 3674 heartbeater.cc:511] Master 127.1.253.126:36927 requested a full tablet report, sending...
I20260924 16:11:44.092782 3182 ts_manager.cc:194] Registered new tserver with Master: 4820bd99d2c74ebea2b72e51cf262902 (127.1.253.67:45585)
I20260924 16:11:44.093513 3182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:57471
I20260924 16:11:44.100658 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.093535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:36927 (local address 127.1.253.67:34095)
0924 16:11:44.094107 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.094123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.094215 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.094568 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.094572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.094772 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:44.094879 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.094886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.095623 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.095626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.096295 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.096305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.096683 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.096686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.096751 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.097204 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:44.097215 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:44.097503 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:44.099318 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:44.099320 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:44.099322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:44.099499 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:44.099851 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:44.099854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:44.099855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:44.099928 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:44.100268 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:44.100270 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:44.100341 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.100416 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.100514 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":306,"threads_started":1}
I20260924 16:11:44.100664 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.093614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:34095 (local address 127.1.253.126:36927)
0924 16:11:44.093728 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.093732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.094180 (+ 448us) server_negotiation.cc:407] Connection header received
0924 16:11:44.094328 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.094331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.094380 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.094461 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:44.095002 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.095493 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.096397 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.096591 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.096663 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.097632 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:44.097635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:44.097637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:44.097657 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:44.099227 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:44.099607 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:44.099611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:44.099612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:44.099754 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:44.100009 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:44.100011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:44.100012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:44.100185 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:44.100265 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.100466 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.100555 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 16:11:44.104928 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:11:44 dist-test-slave-6wvm krb5kdc[3137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:11:44 dist-test-slave-6wvm krb5kdc[3137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:44.131531 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.122798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.1.253.126:36927)
0924 16:11:44.122970 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.122974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.122990 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:11:44.123040 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.123042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.123087 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.123216 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:44.124027 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.124567 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.125445 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.125673 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.125807 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.128263 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:44.128267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:44.128269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:44.128294 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:44.129750 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:44.130273 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:44.130277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:44.130279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:44.130391 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:44.130692 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:44.130695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:44.130696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:44.130886 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:44.130992 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.131232 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.131334 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 16:11:44.134825 3182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:11:44.137791 3182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:11:44.152215 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.146881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.1.253.66:42917)
0924 16:11:44.147250 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.147263 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.147289 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:11:44.147525 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.147532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.147756 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.147937 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:44.148454 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.149554 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.150977 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.151505 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.151637 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.151794 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.151965 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.152053 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":127,"threads_started":1}
I20260924 16:11:44.152201 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.146746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:42917 (local address 127.0.0.1:44606)
0924 16:11:44.147120 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.147180 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.147317 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.148133 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.148142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.148171 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:44.148317 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.148328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.149657 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.149663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.150860 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.150869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.151638 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.151641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.151738 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.151773 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.151931 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.152053 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20260924 16:11:44.153738 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.147843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:34819 (local address 127.0.0.1:40778)
0924 16:11:44.148350 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.148364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.148454 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.149020 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.149023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.149041 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:44.149207 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.149220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.150693 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.150698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.152045 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.152056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.153261 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.153264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.153352 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.153421 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.153492 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.153581 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":159,"spinlock_wait_cycles":2432,"thread_start_us":94,"threads_started":1}
I20260924 16:11:44.153813 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.147710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.1.253.67:45585)
0924 16:11:44.148114 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.148125 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.148354 (+ 229us) server_negotiation.cc:407] Connection header received
0924 16:11:44.148591 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.148596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.148781 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.148913 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:44.149404 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.150541 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.152196 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.152928 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.153013 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.153195 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.153571 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.153660 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":90,"threads_started":1}
I20260924 16:11:44.154147 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.147525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:45585 (local address 127.0.0.1:34634)
0924 16:11:44.148261 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.148274 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.148437 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.149068 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.149070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.149097 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:44.149251 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.149257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.150709 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.150712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.152045 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.152056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.153302 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.153305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.153427 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.153448 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.153939 (+ 491us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.154012 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":159,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":1}
I20260924 16:11:44.154186 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.148261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.1.253.65:34819)
0924 16:11:44.148633 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.148638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.148655 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:11:44.148715 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.148718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.148859 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.149048 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:44.149325 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.150541 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.152196 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.152928 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.153045 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.153179 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.153974 (+ 795us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.154063 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":70,"threads_started":1}
I20260924 16:11:44.156132 3465 tablet_service.cc:1467] Processing CreateTablet for tablet 6746c32a2ffe4619a8bf6886ca9395c3 (DEFAULT_TABLE table=test-table [id=d8b16592729d4f18a1d3613a3c6b05bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:44.156342 3324 tablet_service.cc:1467] Processing CreateTablet for tablet 6746c32a2ffe4619a8bf6886ca9395c3 (DEFAULT_TABLE table=test-table [id=d8b16592729d4f18a1d3613a3c6b05bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:44.156778 3607 tablet_service.cc:1467] Processing CreateTablet for tablet 6746c32a2ffe4619a8bf6886ca9395c3 (DEFAULT_TABLE table=test-table [id=d8b16592729d4f18a1d3613a3c6b05bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:44.157307 3465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6746c32a2ffe4619a8bf6886ca9395c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:44.157307 3324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6746c32a2ffe4619a8bf6886ca9395c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:44.157721 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6746c32a2ffe4619a8bf6886ca9395c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:44.163812 3697 tablet_bootstrap.cc:492] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc: Bootstrap starting.
I20260924 16:11:44.165962 3698 tablet_bootstrap.cc:492] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902: Bootstrap starting.
I20260924 16:11:44.166311 3697 tablet_bootstrap.cc:654] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:44.166389 3699 tablet_bootstrap.cc:492] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17: Bootstrap starting.
I20260924 16:11:44.167117 3697 log.cc:824] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:44.168063 3698 tablet_bootstrap.cc:654] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:44.168406 3699 tablet_bootstrap.cc:654] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:44.168900 3698 log.cc:824] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:44.169163 3697 tablet_bootstrap.cc:492] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc: No bootstrap required, opened a new log
I20260924 16:11:44.169186 3699 log.cc:824] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:44.169370 3697 ts_tablet_manager.cc:1397] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:11:44.170836 3698 tablet_bootstrap.cc:492] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902: No bootstrap required, opened a new log
I20260924 16:11:44.171022 3698 ts_tablet_manager.cc:1397] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:11:44.171072 3699 tablet_bootstrap.cc:492] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17: No bootstrap required, opened a new log
I20260924 16:11:44.171219 3699 ts_tablet_manager.cc:1397] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:11:44.173074 3697 raft_consensus.cc:348] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } }
I20260924 16:11:44.173301 3697 raft_consensus.cc:374] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:44.173359 3697 raft_consensus.cc:729] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271109896a3c4443bf1e4bcca45699fc, State: Initialized, Role: FOLLOWER
I20260924 16:11:44.173807 3697 consensus_queue.cc:260] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } }
I20260924 16:11:44.174820 3697 ts_tablet_manager.cc:1428] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:11:44.174785 3699 raft_consensus.cc:348] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } }
I20260924 16:11:44.174785 3698 raft_consensus.cc:348] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } }
I20260924 16:11:44.175014 3698 raft_consensus.cc:374] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:44.175084 3698 raft_consensus.cc:729] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4820bd99d2c74ebea2b72e51cf262902, State: Initialized, Role: FOLLOWER
I20260924 16:11:44.175203 3391 heartbeater.cc:503] Master 127.1.253.126:36927 was elected leader, sending a full tablet report...
I20260924 16:11:44.175534 3698 consensus_queue.cc:260] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } }
I20260924 16:11:44.175014 3699 raft_consensus.cc:374] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:44.175830 3699 raft_consensus.cc:729] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159a61e457e8439c9c2c96d318633c17, State: Initialized, Role: FOLLOWER
I20260924 16:11:44.176280 3699 consensus_queue.cc:260] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } }
I20260924 16:11:44.176486 3698 ts_tablet_manager.cc:1428] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:11:44.176960 3531 heartbeater.cc:503] Master 127.1.253.126:36927 was elected leader, sending a full tablet report...
I20260924 16:11:44.177250 3699 ts_tablet_manager.cc:1428] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:11:44.177610 3674 heartbeater.cc:503] Master 127.1.253.126:36927 was elected leader, sending a full tablet report...
W20260924 16:11:44.326929 3532 tablet.cc:2369] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:11:44.336308 3675 tablet.cc:2369] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:11:44.346161 3392 tablet.cc:2369] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:44.408663 3703 raft_consensus.cc:482] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:44.408883 3703 raft_consensus.cc:504] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } }
I20260924 16:11:44.409371 3704 raft_consensus.cc:482] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:44.409600 3704 raft_consensus.cc:504] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } }
I20260924 16:11:44.410620 3703 leader_election.cc:288] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159a61e457e8439c9c2c96d318633c17 (127.1.253.66:42917), 4820bd99d2c74ebea2b72e51cf262902 (127.1.253.67:45585)
I20260924 16:11:44.411638 3704 leader_election.cc:288] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159a61e457e8439c9c2c96d318633c17 (127.1.253.66:42917), 271109896a3c4443bf1e4bcca45699fc (127.1.253.65:34819)
I20260924 16:11:44.415525 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.410982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:42917 (local address 127.1.253.67:33907)
0924 16:11:44.411183 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.411199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.411307 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.411644 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.411649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.411675 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:44.411939 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.411950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.413035 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.413038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.414385 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.414395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.415025 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.415028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.415197 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.415221 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.415300 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.415377 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 16:11:44.416006 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.411023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:33907 (local address 127.1.253.66:42917)
0924 16:11:44.411341 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.411347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.411382 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:11:44.411446 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.411452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.411512 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.411599 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:44.412104 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.412877 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.414501 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.414899 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.415033 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.415088 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.415799 (+ 711us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.415890 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260924 16:11:44.416693 3485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c32a2ffe4619a8bf6886ca9395c3" candidate_uuid: "4820bd99d2c74ebea2b72e51cf262902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159a61e457e8439c9c2c96d318633c17" is_pre_election: true
I20260924 16:11:44.417034 3485 raft_consensus.cc:2454] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4820bd99d2c74ebea2b72e51cf262902 in term 0.
I20260924 16:11:44.417160 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.411899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:33261 (local address 127.1.253.65:34819)
0924 16:11:44.412013 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.412018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.412095 (+ 77us) server_negotiation.cc:407] Connection header received
0924 16:11:44.412263 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.412266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.412311 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.412356 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:44.413259 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.414029 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.415495 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.416155 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.416286 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.416339 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.416996 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.417051 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":33}
I20260924 16:11:44.417585 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.411021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:44133 (local address 127.1.253.66:42917)
0924 16:11:44.411531 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.411536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.412234 (+ 698us) server_negotiation.cc:407] Connection header received
0924 16:11:44.412283 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.412289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.412345 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.412466 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:44.412848 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.413649 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.415407 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.416214 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.416339 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.416452 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.417230 (+ 778us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.417314 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":426,"thread_start_us":95,"threads_started":1}
I20260924 16:11:44.417641 3559 leader_election.cc:302] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159a61e457e8439c9c2c96d318633c17, 4820bd99d2c74ebea2b72e51cf262902; no voters:
I20260924 16:11:44.417182 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.410944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:42917 (local address 127.1.253.65:44133)
0924 16:11:44.412026 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.412041 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.412142 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.412581 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.412584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.412602 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:44.412721 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.412727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.413782 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.413785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.415246 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.415257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.416597 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.416601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.416842 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.416870 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.416939 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.417019 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":182,"thread_start_us":121,"threads_started":1}
I20260924 16:11:44.418001 3704 raft_consensus.cc:2781] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:44.418109 3704 raft_consensus.cc:482] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:44.418161 3704 raft_consensus.cc:3037] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:44.418295 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.411389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:45585 (local address 127.1.253.65:45775)
0924 16:11:44.412136 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.412148 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.412267 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.412611 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.412613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.412636 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:44.412786 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.412795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.413782 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.413785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.416817 (+ 3032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.416829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.417807 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.417811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.417909 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.417930 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.418038 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.418115 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":1235,"spinlock_wait_cycles":18688,"thread_start_us":75,"threads_started":1}
I20260924 16:11:44.418362 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.411710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:45775 (local address 127.1.253.67:45585)
0924 16:11:44.411842 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.411847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.412208 (+ 361us) server_negotiation.cc:407] Connection header received
0924 16:11:44.412374 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.412377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.412427 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.412474 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:44.412920 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.413648 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.416963 (+ 3315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.417541 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.417657 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.417704 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.418064 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.418145 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59,"spinlock_wait_cycles":10368}
I20260924 16:11:44.418377 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.411369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:34819 (local address 127.1.253.67:33261)
0924 16:11:44.412017 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.412031 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.412157 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.412478 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.412481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.412522 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:44.413102 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.413109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.414133 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.414136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.415365 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.415377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.416584 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.416589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.416745 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.416779 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.417773 (+ 994us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.417946 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":3,"spinlock_wait_cycles":896,"thread_start_us":106,"threads_started":1}
I20260924 16:11:44.418896 3485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c32a2ffe4619a8bf6886ca9395c3" candidate_uuid: "271109896a3c4443bf1e4bcca45699fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159a61e457e8439c9c2c96d318633c17" is_pre_election: true
I20260924 16:11:44.419044 3485 raft_consensus.cc:2454] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 271109896a3c4443bf1e4bcca45699fc in term 0.
I20260924 16:11:44.419013 3344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c32a2ffe4619a8bf6886ca9395c3" candidate_uuid: "4820bd99d2c74ebea2b72e51cf262902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271109896a3c4443bf1e4bcca45699fc" is_pre_election: true
I20260924 16:11:44.419044 3627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c32a2ffe4619a8bf6886ca9395c3" candidate_uuid: "271109896a3c4443bf1e4bcca45699fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4820bd99d2c74ebea2b72e51cf262902" is_pre_election: true
I20260924 16:11:44.419229 3344 raft_consensus.cc:2454] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4820bd99d2c74ebea2b72e51cf262902 in term 0.
I20260924 16:11:44.419500 3276 leader_election.cc:302] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159a61e457e8439c9c2c96d318633c17, 271109896a3c4443bf1e4bcca45699fc; no voters:
I20260924 16:11:44.419759 3704 raft_consensus.cc:504] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } }
I20260924 16:11:44.419803 3703 raft_consensus.cc:2781] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:44.419916 3703 raft_consensus.cc:482] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:44.419979 3703 raft_consensus.cc:3037] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:44.420044 3627 raft_consensus.cc:2379] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 271109896a3c4443bf1e4bcca45699fc in current term 1: Already voted for candidate 4820bd99d2c74ebea2b72e51cf262902 in this term.
I20260924 16:11:44.420702 3704 leader_election.cc:288] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [CANDIDATE]: Term 1 election: Requested vote from peers 159a61e457e8439c9c2c96d318633c17 (127.1.253.66:42917), 271109896a3c4443bf1e4bcca45699fc (127.1.253.65:34819)
I20260924 16:11:44.421142 3344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c32a2ffe4619a8bf6886ca9395c3" candidate_uuid: "4820bd99d2c74ebea2b72e51cf262902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271109896a3c4443bf1e4bcca45699fc"
I20260924 16:11:44.421492 3703 raft_consensus.cc:504] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } }
I20260924 16:11:44.421748 3344 raft_consensus.cc:2379] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4820bd99d2c74ebea2b72e51cf262902 in current term 1: Already voted for candidate 271109896a3c4443bf1e4bcca45699fc in this term.
I20260924 16:11:44.421877 3703 leader_election.cc:288] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [CANDIDATE]: Term 1 election: Requested vote from peers 159a61e457e8439c9c2c96d318633c17 (127.1.253.66:42917), 4820bd99d2c74ebea2b72e51cf262902 (127.1.253.67:45585)
I20260924 16:11:44.422081 3485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c32a2ffe4619a8bf6886ca9395c3" candidate_uuid: "4820bd99d2c74ebea2b72e51cf262902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159a61e457e8439c9c2c96d318633c17"
I20260924 16:11:44.422202 3485 raft_consensus.cc:3037] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:44.422358 3627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c32a2ffe4619a8bf6886ca9395c3" candidate_uuid: "271109896a3c4443bf1e4bcca45699fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4820bd99d2c74ebea2b72e51cf262902"
I20260924 16:11:44.422528 3627 raft_consensus.cc:2379] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 271109896a3c4443bf1e4bcca45699fc in current term 1: Already voted for candidate 4820bd99d2c74ebea2b72e51cf262902 in this term.
I20260924 16:11:44.423381 3485 raft_consensus.cc:2454] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4820bd99d2c74ebea2b72e51cf262902 in term 1.
I20260924 16:11:44.423673 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c32a2ffe4619a8bf6886ca9395c3" candidate_uuid: "271109896a3c4443bf1e4bcca45699fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159a61e457e8439c9c2c96d318633c17"
I20260924 16:11:44.423705 3559 leader_election.cc:302] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 159a61e457e8439c9c2c96d318633c17, 4820bd99d2c74ebea2b72e51cf262902; no voters: 271109896a3c4443bf1e4bcca45699fc
I20260924 16:11:44.423868 3484 raft_consensus.cc:2379] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 271109896a3c4443bf1e4bcca45699fc in current term 1: Already voted for candidate 4820bd99d2c74ebea2b72e51cf262902 in this term.
I20260924 16:11:44.423883 3704 raft_consensus.cc:2781] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:44.424132 3704 raft_consensus.cc:686] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 1 LEADER]: Becoming Leader. State: Replica: 4820bd99d2c74ebea2b72e51cf262902, State: Running, Role: LEADER
I20260924 16:11:44.424271 3276 leader_election.cc:302] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 271109896a3c4443bf1e4bcca45699fc; no voters: 159a61e457e8439c9c2c96d318633c17, 4820bd99d2c74ebea2b72e51cf262902
I20260924 16:11:44.424423 3703 raft_consensus.cc:2726] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 16:11:44.424721 3704 consensus_queue.cc:237] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } }
I20260924 16:11:44.428175 3183 catalog_manager.cc:5495] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4820bd99d2c74ebea2b72e51cf262902 (127.1.253.67). New cstate: current_term: 1 leader_uuid: "4820bd99d2c74ebea2b72e51cf262902" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } health_report { overall_health: UNKNOWN } } }
I20260924 16:11:44.449288 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.445393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.1.253.67:45585)
0924 16:11:44.445559 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.445564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.445578 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:11:44.445834 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.445837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.445891 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.445992 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:44.446288 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.446798 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.447703 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.447922 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.448027 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.448386 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:44.448877 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:44.449008 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.449089 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.449143 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:11:44.455590 3484 raft_consensus.cc:1261] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17 [term 1 FOLLOWER]: Refusing update from remote peer 4820bd99d2c74ebea2b72e51cf262902: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:44.455588 3344 raft_consensus.cc:1261] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 1 FOLLOWER]: Refusing update from remote peer 4820bd99d2c74ebea2b72e51cf262902: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:44.456288 3710 consensus_queue.cc:1035] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:44.456511 3704 consensus_queue.cc:1035] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:44.464506 3713 mvcc.cc:205] Tried to move back new op lower bound from 7332930783042011136 to 7332930782926417920. Current Snapshot: MvccSnapshot[applied={T|T < 7332930783042011136}]
I20260924 16:11:44.465870 3715 mvcc.cc:205] Tried to move back new op lower bound from 7332930783042011136 to 7332930782926417920. Current Snapshot: MvccSnapshot[applied={T|T < 7332930783042011136}]
I20260924 16:11:44.476718 3183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:44.476886 3183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:44.479440 3183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19: Sending DeleteTablet for 3 replicas of tablet 6746c32a2ffe4619a8bf6886ca9395c3
I20260924 16:11:44.480299 3465 tablet_service.cc:1514] Processing DeleteTablet for tablet 6746c32a2ffe4619a8bf6886ca9395c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:44606
I20260924 16:11:44.480933 3607 tablet_service.cc:1514] Processing DeleteTablet for tablet 6746c32a2ffe4619a8bf6886ca9395c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:34634
I20260924 16:11:44.480934 3324 tablet_service.cc:1514] Processing DeleteTablet for tablet 6746c32a2ffe4619a8bf6886ca9395c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:40778
I20260924 16:11:44.481264 3183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:11:44.481523 3724 tablet_replica.cc:330] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc: stopping tablet replica
W20260924 16:11:44.481698 3183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:11:44.481879 3723 tablet_replica.cc:330] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902: stopping tablet replica
I20260924 16:11:44.481874 3724 raft_consensus.cc:2229] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:44.482227 3724 raft_consensus.cc:2258] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:44.482265 3723 raft_consensus.cc:2229] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:11:44.482795 3722 tablet_replica.cc:330] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17: stopping tablet replica
I20260924 16:11:44.483134 3722 raft_consensus.cc:2229] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:44.483456 3722 raft_consensus.cc:2258] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:44.483991 3723 raft_consensus.cc:2258] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:44.484119 3724 ts_tablet_manager.cc:1905] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:44.485495 3722 ts_tablet_manager.cc:1905] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:44.486598 3723 ts_tablet_manager.cc:1905] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:44.487124 3724 ts_tablet_manager.cc:1918] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:11:44.487233 3724 log.cc:1196] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/wal/wals/6746c32a2ffe4619a8bf6886ca9395c3
I20260924 16:11:44.487577 3724 ts_tablet_manager.cc:1939] T 6746c32a2ffe4619a8bf6886ca9395c3 P 271109896a3c4443bf1e4bcca45699fc: Deleting consensus metadata
I20260924 16:11:44.487772 3324 tablet_service.cc:1467] Processing CreateTablet for tablet e14e98ea9c6a4a0a90cdb1ac858bdade (DEFAULT_TABLE table=test-table [id=4aafb2d695584d3396717be499a1d19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:44.488061 3324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14e98ea9c6a4a0a90cdb1ac858bdade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:44.488695 3722 ts_tablet_manager.cc:1918] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:11:44.488822 3722 log.cc:1196] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/6746c32a2ffe4619a8bf6886ca9395c3
I20260924 16:11:44.488875 3169 catalog_manager.cc:4841] TS 271109896a3c4443bf1e4bcca45699fc (127.1.253.65:34819): tablet 6746c32a2ffe4619a8bf6886ca9395c3 (table test-table [id=d8b16592729d4f18a1d3613a3c6b05bb]) successfully deleted
I20260924 16:11:44.489214 3722 ts_tablet_manager.cc:1939] T 6746c32a2ffe4619a8bf6886ca9395c3 P 159a61e457e8439c9c2c96d318633c17: Deleting consensus metadata
I20260924 16:11:44.489254 3723 ts_tablet_manager.cc:1918] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:11:44.489359 3723 log.cc:1196] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/6746c32a2ffe4619a8bf6886ca9395c3
I20260924 16:11:44.489390 3607 tablet_service.cc:1467] Processing CreateTablet for tablet e14e98ea9c6a4a0a90cdb1ac858bdade (DEFAULT_TABLE table=test-table [id=4aafb2d695584d3396717be499a1d19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:44.489668 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14e98ea9c6a4a0a90cdb1ac858bdade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:44.490391 3697 tablet_bootstrap.cc:492] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc: Bootstrap starting.
I20260924 16:11:44.490581 3169 catalog_manager.cc:4841] TS 159a61e457e8439c9c2c96d318633c17 (127.1.253.66:42917): tablet 6746c32a2ffe4619a8bf6886ca9395c3 (table test-table [id=d8b16592729d4f18a1d3613a3c6b05bb]) successfully deleted
I20260924 16:11:44.490715 3465 tablet_service.cc:1467] Processing CreateTablet for tablet e14e98ea9c6a4a0a90cdb1ac858bdade (DEFAULT_TABLE table=test-table [id=4aafb2d695584d3396717be499a1d19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:44.490944 3465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14e98ea9c6a4a0a90cdb1ac858bdade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:44.491403 3697 tablet_bootstrap.cc:654] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:44.492511 3698 tablet_bootstrap.cc:492] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902: Bootstrap starting.
I20260924 16:11:44.492708 3697 tablet_bootstrap.cc:492] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc: No bootstrap required, opened a new log
I20260924 16:11:44.492779 3697 ts_tablet_manager.cc:1397] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:11:44.493203 3697 raft_consensus.cc:348] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.493299 3699 tablet_bootstrap.cc:492] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17: Bootstrap starting.
I20260924 16:11:44.493311 3697 raft_consensus.cc:374] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:44.493343 3697 raft_consensus.cc:729] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271109896a3c4443bf1e4bcca45699fc, State: Initialized, Role: FOLLOWER
I20260924 16:11:44.493462 3698 tablet_bootstrap.cc:654] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:44.493474 3697 consensus_queue.cc:260] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.493706 3697 ts_tablet_manager.cc:1428] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:11:44.494704 3699 tablet_bootstrap.cc:654] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:44.495869 3699 tablet_bootstrap.cc:492] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17: No bootstrap required, opened a new log
I20260924 16:11:44.495949 3699 ts_tablet_manager.cc:1397] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:11:44.496182 3698 tablet_bootstrap.cc:492] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902: No bootstrap required, opened a new log
I20260924 16:11:44.496258 3698 ts_tablet_manager.cc:1397] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 16:11:44.496367 3699 raft_consensus.cc:348] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.496485 3699 raft_consensus.cc:374] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:44.496515 3699 raft_consensus.cc:729] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159a61e457e8439c9c2c96d318633c17, State: Initialized, Role: FOLLOWER
I20260924 16:11:44.496661 3699 consensus_queue.cc:260] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.496734 3698 raft_consensus.cc:348] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.496840 3698 raft_consensus.cc:374] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:44.496877 3698 raft_consensus.cc:729] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4820bd99d2c74ebea2b72e51cf262902, State: Initialized, Role: FOLLOWER
I20260924 16:11:44.497010 3698 consensus_queue.cc:260] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.497285 3698 ts_tablet_manager.cc:1428] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:11:44.489667 3723 ts_tablet_manager.cc:1939] T 6746c32a2ffe4619a8bf6886ca9395c3 P 4820bd99d2c74ebea2b72e51cf262902: Deleting consensus metadata
I20260924 16:11:44.497831 3699 ts_tablet_manager.cc:1428] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:11:44.499058 3169 catalog_manager.cc:4841] TS 4820bd99d2c74ebea2b72e51cf262902 (127.1.253.67:45585): tablet 6746c32a2ffe4619a8bf6886ca9395c3 (table test-table [id=d8b16592729d4f18a1d3613a3c6b05bb]) successfully deleted
I20260924 16:11:44.651504 3703 raft_consensus.cc:482] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:44.651638 3703 raft_consensus.cc:504] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.652032 3703 leader_election.cc:288] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 159a61e457e8439c9c2c96d318633c17 (127.1.253.66:42917), 4820bd99d2c74ebea2b72e51cf262902 (127.1.253.67:45585)
I20260924 16:11:44.652450 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14e98ea9c6a4a0a90cdb1ac858bdade" candidate_uuid: "271109896a3c4443bf1e4bcca45699fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159a61e457e8439c9c2c96d318633c17" is_pre_election: true
I20260924 16:11:44.652521 3627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14e98ea9c6a4a0a90cdb1ac858bdade" candidate_uuid: "271109896a3c4443bf1e4bcca45699fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4820bd99d2c74ebea2b72e51cf262902" is_pre_election: true
I20260924 16:11:44.652655 3484 raft_consensus.cc:2454] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 271109896a3c4443bf1e4bcca45699fc in term 0.
I20260924 16:11:44.652712 3627 raft_consensus.cc:2454] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 271109896a3c4443bf1e4bcca45699fc in term 0.
I20260924 16:11:44.653050 3276 leader_election.cc:302] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 271109896a3c4443bf1e4bcca45699fc, 4820bd99d2c74ebea2b72e51cf262902; no voters:
I20260924 16:11:44.653229 3703 raft_consensus.cc:2781] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:44.653291 3703 raft_consensus.cc:482] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:44.653334 3703 raft_consensus.cc:3037] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:44.654359 3703 raft_consensus.cc:504] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.654770 3703 leader_election.cc:288] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [CANDIDATE]: Term 1 election: Requested vote from peers 159a61e457e8439c9c2c96d318633c17 (127.1.253.66:42917), 4820bd99d2c74ebea2b72e51cf262902 (127.1.253.67:45585)
I20260924 16:11:44.655118 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14e98ea9c6a4a0a90cdb1ac858bdade" candidate_uuid: "271109896a3c4443bf1e4bcca45699fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "159a61e457e8439c9c2c96d318633c17"
I20260924 16:11:44.655246 3484 raft_consensus.cc:3037] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:44.655211 3627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14e98ea9c6a4a0a90cdb1ac858bdade" candidate_uuid: "271109896a3c4443bf1e4bcca45699fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4820bd99d2c74ebea2b72e51cf262902"
I20260924 16:11:44.655372 3627 raft_consensus.cc:3037] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:44.656252 3484 raft_consensus.cc:2454] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 271109896a3c4443bf1e4bcca45699fc in term 1.
I20260924 16:11:44.656327 3627 raft_consensus.cc:2454] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 271109896a3c4443bf1e4bcca45699fc in term 1.
I20260924 16:11:44.656587 3276 leader_election.cc:302] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159a61e457e8439c9c2c96d318633c17, 271109896a3c4443bf1e4bcca45699fc; no voters:
I20260924 16:11:44.656785 3703 raft_consensus.cc:2781] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:44.656975 3703 raft_consensus.cc:686] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 1 LEADER]: Becoming Leader. State: Replica: 271109896a3c4443bf1e4bcca45699fc, State: Running, Role: LEADER
I20260924 16:11:44.657243 3703 consensus_queue.cc:237] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.659631 3182 catalog_manager.cc:5495] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 271109896a3c4443bf1e4bcca45699fc (127.1.253.65). New cstate: current_term: 1 leader_uuid: "271109896a3c4443bf1e4bcca45699fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } health_report { overall_health: UNKNOWN } } }
Sep 24 16:11:44 dist-test-slave-6wvm krb5kdc[3137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:44.716375 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.708060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:36927 (local address 127.0.0.1:55250)
0924 16:11:44.708425 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.708448 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.708561 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.708826 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.708828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.709249 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:44.709355 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.709362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.710177 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.710180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.710765 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.710772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.711234 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.711236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.711297 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.711881 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:44.711892 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:44.713345 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:44.714923 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:44.714932 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:44.714938 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:44.715168 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:44.715406 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:44.715408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:44.715409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:44.715484 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:44.715915 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:44.715921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:44.716024 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.716117 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.716231 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"spinlock_wait_cycles":12032,"thread_start_us":134,"threads_started":1}
I20260924 16:11:44.716560 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.708087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.1.253.126:36927)
0924 16:11:44.708403 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.708411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.708525 (+ 114us) server_negotiation.cc:407] Connection header received
0924 16:11:44.708651 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.708653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.708695 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.708788 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:44.709507 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.710069 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.710899 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.711123 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.711215 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.713476 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:44.713480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:44.713481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:44.713501 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:44.714824 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:44.715252 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:44.715255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:44.715256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:44.715329 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:44.715595 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:44.715598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:44.715599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:44.715802 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:44.715908 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.716210 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.716330 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":134,"threads_started":1}
I20260924 16:11:44.720412 3182 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:55250:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:11:44.721156 3182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:11:44.726473 3324 tablet_service.cc:1467] Processing CreateTablet for tablet 9b39f3311d9643f48703b3fa25cd4378 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91b5dc9a9e0440748d916e17875392d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:11:44.726508 3465 tablet_service.cc:1467] Processing CreateTablet for tablet 9b39f3311d9643f48703b3fa25cd4378 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91b5dc9a9e0440748d916e17875392d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:11:44.726953 3465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b39f3311d9643f48703b3fa25cd4378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:44.726964 3324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b39f3311d9643f48703b3fa25cd4378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:44.729558 3699 tablet_bootstrap.cc:492] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: Bootstrap starting.
I20260924 16:11:44.729566 3697 tablet_bootstrap.cc:492] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc: Bootstrap starting.
I20260924 16:11:44.730446 3607 tablet_service.cc:1467] Processing CreateTablet for tablet 9b39f3311d9643f48703b3fa25cd4378 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91b5dc9a9e0440748d916e17875392d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:11:44.730679 3697 tablet_bootstrap.cc:654] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:44.730691 3699 tablet_bootstrap.cc:654] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:44.730849 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b39f3311d9643f48703b3fa25cd4378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:44.732026 3699 tablet_bootstrap.cc:492] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: No bootstrap required, opened a new log
I20260924 16:11:44.732120 3699 ts_tablet_manager.cc:1397] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:11:44.732705 3699 raft_consensus.cc:348] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.732822 3699 raft_consensus.cc:374] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:44.732854 3699 raft_consensus.cc:729] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159a61e457e8439c9c2c96d318633c17, State: Initialized, Role: FOLLOWER
I20260924 16:11:44.733049 3699 consensus_queue.cc:260] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.733312 3705 tablet_replica.cc:439] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } }
I20260924 16:11:44.733445 3705 tablet_replica.cc:442] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:44.733685 3699 ts_tablet_manager.cc:1428] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:11:44.733824 3698 tablet_bootstrap.cc:492] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902: Bootstrap starting.
I20260924 16:11:44.734730 3698 tablet_bootstrap.cc:654] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:44.734802 3697 tablet_bootstrap.cc:492] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc: No bootstrap required, opened a new log
I20260924 16:11:44.734896 3697 ts_tablet_manager.cc:1397] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 16:11:44.735384 3697 raft_consensus.cc:348] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.735517 3697 raft_consensus.cc:374] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:44.735565 3697 raft_consensus.cc:729] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271109896a3c4443bf1e4bcca45699fc, State: Initialized, Role: FOLLOWER
I20260924 16:11:44.735702 3697 consensus_queue.cc:260] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.736035 3698 tablet_bootstrap.cc:492] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902: No bootstrap required, opened a new log
I20260924 16:11:44.736063 3697 ts_tablet_manager.cc:1428] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 16:11:44.736002 3703 tablet_replica.cc:439] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } }
I20260924 16:11:44.736114 3698 ts_tablet_manager.cc:1397] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 16:11:44.736145 3703 tablet_replica.cc:442] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:44.736524 3698 raft_consensus.cc:348] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.736670 3698 raft_consensus.cc:374] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:44.736717 3698 raft_consensus.cc:729] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4820bd99d2c74ebea2b72e51cf262902, State: Initialized, Role: FOLLOWER
I20260924 16:11:44.736852 3698 consensus_queue.cc:260] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.737128 3698 ts_tablet_manager.cc:1428] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 16:11:44.737110 3704 tablet_replica.cc:439] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } }
I20260924 16:11:44.737308 3704 tablet_replica.cc:442] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:44.806650 3705 raft_consensus.cc:482] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:44.806807 3705 raft_consensus.cc:504] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.807838 3705 leader_election.cc:288] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 271109896a3c4443bf1e4bcca45699fc (127.1.253.65:34819), 4820bd99d2c74ebea2b72e51cf262902 (127.1.253.67:45585)
I20260924 16:11:44.811914 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.808183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:40863 (local address 127.1.253.65:34819)
0924 16:11:44.808322 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.808326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.808474 (+ 148us) server_negotiation.cc:407] Connection header received
0924 16:11:44.808677 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.808681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.808740 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.808830 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:44.809138 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.809670 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.810914 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.811323 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.811488 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.811549 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.811721 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.811794 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:11:44.811931 3741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.808071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:34819 (local address 127.1.253.66:40863)
0924 16:11:44.808370 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.808387 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.808501 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.808869 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.808873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.808893 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:44.809018 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.809024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.809797 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.809801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.810772 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.810782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.811473 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.811477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.811582 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.811612 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.811703 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.811771 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":2,"thread_start_us":124,"threads_started":1}
I20260924 16:11:44.811977 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.808634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:44013 (local address 127.1.253.67:45585)
0924 16:11:44.808759 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.808765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.808908 (+ 143us) server_negotiation.cc:407] Connection header received
0924 16:11:44.809082 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.809085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.809129 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.809208 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:44.809482 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.809977 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.811018 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.811431 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.811562 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.811625 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.811794 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.811867 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:11:44.811997 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.808490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:45585 (local address 127.1.253.66:44013)
0924 16:11:44.808806 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.808818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.808959 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.809230 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.809233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.809248 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:44.809364 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.809369 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.810100 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.810104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.810888 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.810897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.811557 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.811561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.811661 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.811684 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.811777 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.811846 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":23,"spinlock_wait_cycles":2944,"thread_start_us":116,"threads_started":1}
I20260924 16:11:44.812506 3627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b39f3311d9643f48703b3fa25cd4378" candidate_uuid: "159a61e457e8439c9c2c96d318633c17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4820bd99d2c74ebea2b72e51cf262902" is_pre_election: true
I20260924 16:11:44.812685 3627 raft_consensus.cc:2454] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159a61e457e8439c9c2c96d318633c17 in term 0.
I20260924 16:11:44.812707 3343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b39f3311d9643f48703b3fa25cd4378" candidate_uuid: "159a61e457e8439c9c2c96d318633c17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271109896a3c4443bf1e4bcca45699fc" is_pre_election: true
I20260924 16:11:44.812872 3343 raft_consensus.cc:2454] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 159a61e457e8439c9c2c96d318633c17 in term 0.
I20260924 16:11:44.813097 3418 leader_election.cc:302] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159a61e457e8439c9c2c96d318633c17, 4820bd99d2c74ebea2b72e51cf262902; no voters:
I20260924 16:11:44.813371 3705 raft_consensus.cc:2781] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:44.813436 3705 raft_consensus.cc:482] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:44.813473 3705 raft_consensus.cc:3037] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:44.814533 3705 raft_consensus.cc:504] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.814893 3705 leader_election.cc:288] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [CANDIDATE]: Term 1 election: Requested vote from peers 271109896a3c4443bf1e4bcca45699fc (127.1.253.65:34819), 4820bd99d2c74ebea2b72e51cf262902 (127.1.253.67:45585)
I20260924 16:11:44.815260 3627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b39f3311d9643f48703b3fa25cd4378" candidate_uuid: "159a61e457e8439c9c2c96d318633c17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4820bd99d2c74ebea2b72e51cf262902"
I20260924 16:11:44.815260 3343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b39f3311d9643f48703b3fa25cd4378" candidate_uuid: "159a61e457e8439c9c2c96d318633c17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271109896a3c4443bf1e4bcca45699fc"
I20260924 16:11:44.815387 3627 raft_consensus.cc:3037] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:44.815419 3343 raft_consensus.cc:3037] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:44.816543 3627 raft_consensus.cc:2454] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159a61e457e8439c9c2c96d318633c17 in term 1.
I20260924 16:11:44.816543 3343 raft_consensus.cc:2454] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159a61e457e8439c9c2c96d318633c17 in term 1.
I20260924 16:11:44.816860 3418 leader_election.cc:302] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159a61e457e8439c9c2c96d318633c17, 271109896a3c4443bf1e4bcca45699fc; no voters:
I20260924 16:11:44.817140 3705 raft_consensus.cc:2781] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:44.817312 3705 raft_consensus.cc:686] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [term 1 LEADER]: Becoming Leader. State: Replica: 159a61e457e8439c9c2c96d318633c17, State: Running, Role: LEADER
I20260924 16:11:44.817570 3705 consensus_queue.cc:237] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } }
I20260924 16:11:44.818792 3743 tablet_replica.cc:439] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: TxnStatusTablet state changed. Reason: New leader 159a61e457e8439c9c2c96d318633c17. Latest consensus state: current_term: 1 leader_uuid: "159a61e457e8439c9c2c96d318633c17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } }
I20260924 16:11:44.818920 3743 tablet_replica.cc:442] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:11:44.819383 3745 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:11:44.819576 3745 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:11:44.820451 3182 catalog_manager.cc:5495] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 159a61e457e8439c9c2c96d318633c17 (127.1.253.66). New cstate: current_term: 1 leader_uuid: "159a61e457e8439c9c2c96d318633c17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } health_report { overall_health: UNKNOWN } } }
I20260924 16:11:44.833104 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.828834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:42917 (local address 127.0.0.1:44610)
0924 16:11:44.829000 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.829019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.829133 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.829511 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.829515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.829522 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:11:44.829638 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.829643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.830533 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.830536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.831333 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.831350 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.831922 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.831925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.832015 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.832074 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:11:44.832788 (+ 714us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:11:44.832806 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.832876 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.832966 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 16:11:44.833468 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.829088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.1.253.66:42917)
0924 16:11:44.829218 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.829222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.829237 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:11:44.829317 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.829319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.829370 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.829481 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:44.829744 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.830400 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.831462 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.831799 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.831918 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.832208 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:44.832694 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:44.832813 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.833282 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.833350 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:11:44.836371 3343 raft_consensus.cc:1261] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc [term 1 FOLLOWER]: Refusing update from remote peer 159a61e457e8439c9c2c96d318633c17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:44.836947 3705 consensus_queue.cc:1035] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:44.837020 3627 raft_consensus.cc:1261] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902 [term 1 FOLLOWER]: Refusing update from remote peer 159a61e457e8439c9c2c96d318633c17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:44.837532 3705 consensus_queue.cc:1035] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:44.839231 3703 tablet_replica.cc:439] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc: TxnStatusTablet state changed. Reason: New leader 159a61e457e8439c9c2c96d318633c17. Latest consensus state: current_term: 1 leader_uuid: "159a61e457e8439c9c2c96d318633c17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } }
I20260924 16:11:44.839371 3703 tablet_replica.cc:442] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:44.840798 3704 tablet_replica.cc:439] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902: TxnStatusTablet state changed. Reason: New leader 159a61e457e8439c9c2c96d318633c17. Latest consensus state: current_term: 1 leader_uuid: "159a61e457e8439c9c2c96d318633c17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } }
I20260924 16:11:44.840934 3704 tablet_replica.cc:442] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:44.842571 3705 tablet_replica.cc:439] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "159a61e457e8439c9c2c96d318633c17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } }
I20260924 16:11:44.842693 3705 tablet_replica.cc:442] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:11:44.843294 3714 mvcc.cc:205] Tried to move back new op lower bound from 7332930784604188672 to 7332930784534990848. Current Snapshot: MvccSnapshot[applied={T|T < 7332930784604188672 or (T in {7332930784604188672})}]
I20260924 16:11:44.844372 3704 tablet_replica.cc:439] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "159a61e457e8439c9c2c96d318633c17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } }
I20260924 16:11:44.844489 3704 tablet_replica.cc:442] T 9b39f3311d9643f48703b3fa25cd4378 P 4820bd99d2c74ebea2b72e51cf262902: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:44.850862 3743 tablet_replica.cc:439] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "159a61e457e8439c9c2c96d318633c17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } }
I20260924 16:11:44.850979 3743 tablet_replica.cc:442] T 9b39f3311d9643f48703b3fa25cd4378 P 159a61e457e8439c9c2c96d318633c17: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:11:44.851954 3703 tablet_replica.cc:439] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "159a61e457e8439c9c2c96d318633c17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271109896a3c4443bf1e4bcca45699fc" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 34819 } } peers { permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 } } peers { permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 } } }
I20260924 16:11:44.852072 3703 tablet_replica.cc:442] T 9b39f3311d9643f48703b3fa25cd4378 P 271109896a3c4443bf1e4bcca45699fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:44.855085 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.849972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.1.253.65:34819)
0924 16:11:44.850089 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.850093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.850105 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:11:44.850186 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.850189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.850278 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.850377 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:44.850750 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.851535 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.853417 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.853777 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.853874 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.854221 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:44.854722 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:44.854813 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.854939 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.854988 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:11:44.867441 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.862875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:35785 (local address 127.1.253.66:42917)
0924 16:11:44.863099 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.863103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.863300 (+ 197us) server_negotiation.cc:407] Connection header received
0924 16:11:44.863444 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.863446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.863489 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.863560 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:44.864011 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.864556 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.866277 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.866506 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.866627 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.867027 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:44.867100 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:44.867155 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.867268 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.867312 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:11:44.867707 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.862796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:42917 (local address 127.1.253.65:35785)
0924 16:11:44.863211 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.863227 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.863334 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.863600 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.863603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.863631 (+ 28us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:11:44.863899 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.863907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.864681 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.864685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.866020 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.866045 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.866656 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.866660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.866820 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.866889 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:11:44.867182 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:11:44.867189 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.867262 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.867347 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":122,"threads_started":1}
I20260924 16:11:44.874984 3627 raft_consensus.cc:1261] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902 [term 1 FOLLOWER]: Refusing update from remote peer 271109896a3c4443bf1e4bcca45699fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:44.875191 3484 raft_consensus.cc:1261] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17 [term 1 FOLLOWER]: Refusing update from remote peer 271109896a3c4443bf1e4bcca45699fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:44.875427 3703 consensus_queue.cc:1035] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4820bd99d2c74ebea2b72e51cf262902" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:44.875655 3729 consensus_queue.cc:1035] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "159a61e457e8439c9c2c96d318633c17" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:44.900929 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.896338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:36045 (local address 127.1.253.65:34819)
0924 16:11:44.896478 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.896482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.896811 (+ 329us) server_negotiation.cc:407] Connection header received
0924 16:11:44.896985 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.896988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.897035 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.897116 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:44.897602 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.898314 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.899730 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.900004 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.900092 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.900441 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:44.900530 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:44.900586 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.900726 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.900784 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:11:44.901393 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.896326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:34819 (local address 127.1.253.66:36045)
0924 16:11:44.896726 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:44.896739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:44.896857 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:44.897136 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:44.897139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:44.897146 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:11:44.897487 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.897495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.898600 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.898603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.899621 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:44.899633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.900113 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.900117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:44.900233 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.900289 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:11:44.900614 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:11:44.900621 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:11:44.900690 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 16:11:44.900772 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":174,"threads_started":1}
I20260924 16:11:44.929563 3182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:44.929780 3182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:44.932214 3182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a06e995408b4af0ae34ce41e0d88b19: Sending DeleteTablet for 3 replicas of tablet e14e98ea9c6a4a0a90cdb1ac858bdade
I20260924 16:11:44.933187 3324 tablet_service.cc:1514] Processing DeleteTablet for tablet e14e98ea9c6a4a0a90cdb1ac858bdade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:40778
I20260924 16:11:44.933298 3607 tablet_service.cc:1514] Processing DeleteTablet for tablet e14e98ea9c6a4a0a90cdb1ac858bdade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:34634
I20260924 16:11:44.933404 3724 tablet_replica.cc:330] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc: stopping tablet replica
I20260924 16:11:44.933494 3723 tablet_replica.cc:330] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902: stopping tablet replica
I20260924 16:11:44.933537 3724 raft_consensus.cc:2229] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:11:44.933604 3723 raft_consensus.cc:2229] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:44.933723 3723 raft_consensus.cc:2258] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:44.933837 3724 raft_consensus.cc:2258] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:44.934636 3724 ts_tablet_manager.cc:1905] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:44.934656 3723 ts_tablet_manager.cc:1905] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:44.935154 3465 tablet_service.cc:1514] Processing DeleteTablet for tablet e14e98ea9c6a4a0a90cdb1ac858bdade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:44606
I20260924 16:11:44.935338 3722 tablet_replica.cc:330] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17: stopping tablet replica
I20260924 16:11:44.935456 3722 raft_consensus.cc:2229] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:44.935562 3722 raft_consensus.cc:2258] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:44.936323 3722 ts_tablet_manager.cc:1905] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:44.937536 3724 ts_tablet_manager.cc:1918] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:11:44.937565 3723 ts_tablet_manager.cc:1918] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:11:44.937603 3724 log.cc:1196] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-0/wal/wals/e14e98ea9c6a4a0a90cdb1ac858bdade
I20260924 16:11:44.937639 3723 log.cc:1196] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/e14e98ea9c6a4a0a90cdb1ac858bdade
I20260924 16:11:44.937891 3724 ts_tablet_manager.cc:1939] T e14e98ea9c6a4a0a90cdb1ac858bdade P 271109896a3c4443bf1e4bcca45699fc: Deleting consensus metadata
I20260924 16:11:44.937902 3723 ts_tablet_manager.cc:1939] T e14e98ea9c6a4a0a90cdb1ac858bdade P 4820bd99d2c74ebea2b72e51cf262902: Deleting consensus metadata
I20260924 16:11:44.938620 3169 catalog_manager.cc:4841] TS 4820bd99d2c74ebea2b72e51cf262902 (127.1.253.67:45585): tablet e14e98ea9c6a4a0a90cdb1ac858bdade (table test-table [id=4aafb2d695584d3396717be499a1d19e]) successfully deleted
I20260924 16:11:44.938778 3169 catalog_manager.cc:4841] TS 271109896a3c4443bf1e4bcca45699fc (127.1.253.65:34819): tablet e14e98ea9c6a4a0a90cdb1ac858bdade (table test-table [id=4aafb2d695584d3396717be499a1d19e]) successfully deleted
I20260924 16:11:44.939172 3722 ts_tablet_manager.cc:1918] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:11:44.939245 3722 log.cc:1196] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/e14e98ea9c6a4a0a90cdb1ac858bdade
I20260924 16:11:44.939497 3722 ts_tablet_manager.cc:1939] T e14e98ea9c6a4a0a90cdb1ac858bdade P 159a61e457e8439c9c2c96d318633c17: Deleting consensus metadata
I20260924 16:11:44.940163 3169 catalog_manager.cc:4841] TS 159a61e457e8439c9c2c96d318633c17 (127.1.253.66:42917): tablet e14e98ea9c6a4a0a90cdb1ac858bdade (table test-table [id=4aafb2d695584d3396717be499a1d19e]) successfully deleted
Sep 24 16:11:44 dist-test-slave-6wvm krb5kdc[3137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.253.65@KRBTEST.COM
I20260924 16:11:44.953408 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.940363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.1.253.65:34819)
0924 16:11:44.940530 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.940535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.940969 (+ 434us) server_negotiation.cc:407] Connection header received
0924 16:11:44.941091 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.941094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.941144 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.941218 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:44.942156 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.943175 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.944672 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.945013 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.945122 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.949088 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:44.949097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:44.949113 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:44.949140 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:44.951354 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:44.951892 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:44.951899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:44.951901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:44.951994 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:44.952297 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:44.952300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:44.952302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:44.952601 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:44.952813 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.953071 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.953240 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 16:11:44.953864 3385 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40802
I20260924 16:11:44.965616 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:44.955783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.1.253.126:36927)
0924 16:11:44.955926 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:44.955931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:44.956251 (+ 320us) server_negotiation.cc:407] Connection header received
0924 16:11:44.956410 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:44.956413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:44.956468 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:44.956553 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:44.957317 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.958178 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.959360 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:44.959766 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:44.959879 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:44.961448 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:44.961452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:44.961455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:44.961485 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:44.963404 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:44.963915 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:44.963919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:44.963921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:44.964018 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:44.964515 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:44.964519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:44.964521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:44.964749 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:44.964909 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:44.965305 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 16:11:44.965452 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 16:11:44.966444 3182 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55252
I20260924 16:11:44.968078 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 3258
I20260924 16:11:44.976328 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 3400
I20260924 16:11:44.984453 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 3540
I20260924 16:11:44.991897 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 3151
2026-09-24T16:11:44Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3052 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:45 dist-test-slave-6wvm krb5kdc[3768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:45 dist-test-slave-6wvm krb5kdc[3768](info): set up 2 sockets
Sep 24 16:11:45 dist-test-slave-6wvm krb5kdc[3768](info): commencing operation
krb5kdc: starting...
W20260924 16:11:46.033480 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:11:46 dist-test-slave-6wvm krb5kdc[3768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:11:46.157299 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:42659
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:42659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:46.274121 3781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:46.274473 3781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:46.274519 3781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:46.278645 3781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:46.278703 3781 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:11:46.278720 3781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:46.278739 3781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:46.278756 3781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:46.278807 3781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:46.284257 3781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:42659
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:42659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:46.285351 3781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:46.286496 3781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:46.294981 3787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:46.294996 3786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:46.295109 3781 server_base.cc:1034] running on GCE node
W20260924 16:11:46.295159 3789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:46.295863 3781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:46.296928 3781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:46.298096 3781 hybrid_clock.cc:648] HybridClock initialized: now 1790266306298077 us; error 29 us; skew 500 ppm
Sep 24 16:11:46 dist-test-slave-6wvm krb5kdc[3768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266306, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:46.300984 3781 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:11:46.301857 3781 webserver.cc:460] Webserver started at http://127.1.253.126:43821/ using document root <none> and password file <none>
I20260924 16:11:46.302464 3781 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:46.302527 3781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:46.302783 3781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:46.304833 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "1e6cb9a2beac456fb289b247b4edfb66"
format_stamp: "Formatted at 2026-09-24 16:11:46 on dist-test-slave-6wvm"
server_key: "af7811c11ab800acb132355e211100b8"
server_key_iv: "dc42f2b58d4ab59aa745099abae48323"
server_key_version: "encryptionkey@0"
I20260924 16:11:46.305398 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "1e6cb9a2beac456fb289b247b4edfb66"
format_stamp: "Formatted at 2026-09-24 16:11:46 on dist-test-slave-6wvm"
server_key: "af7811c11ab800acb132355e211100b8"
server_key_iv: "dc42f2b58d4ab59aa745099abae48323"
server_key_version: "encryptionkey@0"
I20260924 16:11:46.309055 3781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 16:11:46.311527 3796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:46.312758 3781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:11:46.312896 3781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "1e6cb9a2beac456fb289b247b4edfb66"
format_stamp: "Formatted at 2026-09-24 16:11:46 on dist-test-slave-6wvm"
server_key: "af7811c11ab800acb132355e211100b8"
server_key_iv: "dc42f2b58d4ab59aa745099abae48323"
server_key_version: "encryptionkey@0"
I20260924 16:11:46.313006 3781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:46.340833 3781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:46.343753 3781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:46.343956 3781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:46.351778 3859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:42659 every 8 connection(s)
I20260924 16:11:46.351859 3781 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:42659
I20260924 16:11:46.352751 3781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:46.353521 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 3781
I20260924 16:11:46.353652 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:11:46.353950 2037 external_mini_cluster.cc:1428] Setting key 85523beb30922a869b181f740b3b2a92
I20260924 16:11:46.356158 3860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:46.362701 3860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66: Bootstrap starting.
Sep 24 16:11:46 dist-test-slave-6wvm krb5kdc[3768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:46.364801 3860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:46.365903 3860 log.cc:824] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:46.367865 3860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66: No bootstrap required, opened a new log
I20260924 16:11:46.368317 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:46.355312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.1.253.126:42659)
0924 16:11:46.355731 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:46.355747 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:46.355782 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:11:46.356484 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:46.356502 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:46.356826 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:46.357097 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:46.357960 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.358986 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.360204 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.360648 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.360759 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:46.363540 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:46.363546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:46.363558 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:46.363592 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:46.365325 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:46.365771 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:46.365777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:46.365781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:46.365890 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:46.366194 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:46.366198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:46.366200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:46.366763 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:46.366961 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:46.367303 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 16:11:46.367565 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
I20260924 16:11:46.370846 3860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6cb9a2beac456fb289b247b4edfb66" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42659 } }
I20260924 16:11:46.371089 3860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:46.371177 3860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6cb9a2beac456fb289b247b4edfb66, State: Initialized, Role: FOLLOWER
I20260924 16:11:46.371618 3860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6cb9a2beac456fb289b247b4edfb66" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42659 } }
I20260924 16:11:46.371771 3860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:46.371861 3860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:46.371976 3860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:46.372951 3860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6cb9a2beac456fb289b247b4edfb66" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42659 } }
I20260924 16:11:46.373308 3860 leader_election.cc:302] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e6cb9a2beac456fb289b247b4edfb66; no voters:
I20260924 16:11:46.373637 3860 leader_election.cc:288] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:46.373708 3866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:46.373945 3866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [term 1 LEADER]: Becoming Leader. State: Replica: 1e6cb9a2beac456fb289b247b4edfb66, State: Running, Role: LEADER
I20260924 16:11:46.374449 3866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6cb9a2beac456fb289b247b4edfb66" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42659 } }
I20260924 16:11:46.374907 3860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:46.376092 3868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e6cb9a2beac456fb289b247b4edfb66. Latest consensus state: current_term: 1 leader_uuid: "1e6cb9a2beac456fb289b247b4edfb66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6cb9a2beac456fb289b247b4edfb66" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42659 } } }
I20260924 16:11:46.376088 3867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e6cb9a2beac456fb289b247b4edfb66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6cb9a2beac456fb289b247b4edfb66" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42659 } } }
I20260924 16:11:46.376241 3867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:46.376241 3868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:46.376672 3875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:46.379832 3875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:46.384934 3875 catalog_manager.cc:1347] Generated new cluster ID: 296f37ed364a4b2681fb421ae9c08c8b
I20260924 16:11:46.384999 3875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:46.413923 3875 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:46.414996 3875 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:46.424924 3875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66: Generated new TSK 0
I20260924 16:11:46.425812 3875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:11:46.473642 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:42659
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:46.588209 3887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:46.588467 3887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:46.588512 3887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:46.592626 3887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:46.592695 3887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:46.592736 3887 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 16:11:46.592749 3887 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:11:46.592819 3887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:11:46.598205 3887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.253.126:42659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:46.599400 3887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:46.600508 3887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:46.610420 3892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:46.610316 3895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:46.610483 3887 server_base.cc:1034] running on GCE node
W20260924 16:11:46.610316 3893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:46.611090 3887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:46.611740 3887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:46.613037 3887 hybrid_clock.cc:648] HybridClock initialized: now 1790266306613023 us; error 26 us; skew 500 ppm
Sep 24 16:11:46 dist-test-slave-6wvm krb5kdc[3768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266306, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:46.615922 3887 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:11:46.616870 3887 webserver.cc:460] Webserver started at http://127.1.253.65:39255/ using document root <none> and password file <none>
I20260924 16:11:46.617520 3887 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:46.617587 3887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:46.617758 3887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:46.619827 3887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "0603fc7358f84df0bef56cf51fb760b6"
format_stamp: "Formatted at 2026-09-24 16:11:46 on dist-test-slave-6wvm"
server_key: "bef426abe37180c183666c4d2c635c13"
server_key_iv: "634fddc127b859543cfb821049e65072"
server_key_version: "encryptionkey@0"
I20260924 16:11:46.620349 3887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "0603fc7358f84df0bef56cf51fb760b6"
format_stamp: "Formatted at 2026-09-24 16:11:46 on dist-test-slave-6wvm"
server_key: "bef426abe37180c183666c4d2c635c13"
server_key_iv: "634fddc127b859543cfb821049e65072"
server_key_version: "encryptionkey@0"
I20260924 16:11:46.624066 3887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260924 16:11:46.626504 3902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:46.627733 3887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:11:46.627830 3887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "0603fc7358f84df0bef56cf51fb760b6"
format_stamp: "Formatted at 2026-09-24 16:11:46 on dist-test-slave-6wvm"
server_key: "bef426abe37180c183666c4d2c635c13"
server_key_iv: "634fddc127b859543cfb821049e65072"
server_key_version: "encryptionkey@0"
I20260924 16:11:46.627915 3887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:46.641943 3887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:46.644896 3887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:46.645051 3887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:46.646538 3887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:46.646592 3887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:46.646659 3887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:46.646692 3887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:11:46 dist-test-slave-6wvm krb5kdc[3768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266306, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:46.657787 3887 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:40767
I20260924 16:11:46.657850 4019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:40767 every 8 connection(s)
I20260924 16:11:46.658771 3887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:11:46.660795 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:46.647386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:38269 (local address 127.1.253.126:42659)
0924 16:11:46.647597 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:46.647605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:46.648417 (+ 812us) server_negotiation.cc:407] Connection header received
0924 16:11:46.649423 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:46.649427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:46.649483 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:46.649544 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:46.651138 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.652052 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.653603 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.654032 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.654169 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:46.657034 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:46.657040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:46.657047 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:46.657084 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:46.658770 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:46.659326 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:46.659330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:46.659331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:46.659420 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:46.659785 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:46.659788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:46.659790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:46.660003 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:46.660105 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:46.660550 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 16:11:46.660673 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:11:46.661312 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 3887
I20260924 16:11:46.661434 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:11:46.661685 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:46.647781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42659 (local address 127.1.253.65:38269)
0924 16:11:46.648237 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:46.648289 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:46.649157 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:46.649925 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:46.649931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:46.650323 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:46.650972 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:46.650996 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.652219 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.652222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:46.653417 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:46.653427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.654228 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.654232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:46.654484 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:46.655159 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:46.655176 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:46.656846 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:46.658895 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:46.658901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:46.658916 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:46.659178 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:46.659525 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:46.659528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:46.659530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:46.659681 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:46.660112 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:46.660117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:46.660344 (+ 227us) client_negotiation.cc:769] Sending connection context
0924 16:11:46.660499 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 16:11:46.660743 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":124,"threads_started":1}
I20260924 16:11:46.661703 2037 external_mini_cluster.cc:1428] Setting key 94de0c81c95baaeba94c466706497639
I20260924 16:11:46.663183 4020 heartbeater.cc:347] Connected to a master server at 127.1.253.126:42659
I20260924 16:11:46.663333 4020 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:46.663882 4020 heartbeater.cc:511] Master 127.1.253.126:42659 requested a full tablet report, sending...
I20260924 16:11:46.665643 3812 ts_manager.cc:194] Registered new tserver with Master: 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767)
I20260924 16:11:46.667253 3812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:38269
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:11:46.672902 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:46.665725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:59353 (local address 127.1.253.126:42659)
0924 16:11:46.665849 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:46.665855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:46.666074 (+ 219us) server_negotiation.cc:407] Connection header received
0924 16:11:46.666161 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:46.666165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:46.666218 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:46.666329 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:46.666781 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.667300 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.668279 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.668536 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.668653 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:46.669874 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:46.669877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:46.669880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:46.669905 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:46.671324 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:46.671733 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:46.671737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:46.671739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:46.671818 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:46.672094 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:46.672097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:46.672099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:46.672264 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:46.672369 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:46.672616 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 16:11:46.672741 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:11:46.672914 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:46.665627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42659 (local address 127.1.253.65:59353)
0924 16:11:46.665967 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:46.665982 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:46.666086 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:46.666343 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:46.666346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:46.666560 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:46.666672 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:46.666679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.667424 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.667427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:46.668139 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:46.668150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.668753 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.668756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:46.668852 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:46.669398 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:46.669409 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:46.669727 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:46.671435 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:46.671440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:46.671442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:46.671637 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:46.671910 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:46.671914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:46.671916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:46.671998 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:46.672359 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:46.672362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:46.672455 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 16:11:46.672560 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 16:11:46.672706 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":8,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:11:46.694566 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:42659
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:46.807763 4030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:46.808035 4030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:46.808112 4030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:46.812258 4030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:46.812346 4030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:46.812422 4030 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 16:11:46.812466 4030 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:11:46.812525 4030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:11:46.818064 4030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.253.126:42659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:46.819211 4030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:46.820298 4030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:46.830331 4036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:46.830349 4038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:46.830401 4030 server_base.cc:1034] running on GCE node
W20260924 16:11:46.830359 4035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:46.831039 4030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:46.831691 4030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:46.832922 4030 hybrid_clock.cc:648] HybridClock initialized: now 1790266306832891 us; error 44 us; skew 500 ppm
Sep 24 16:11:46 dist-test-slave-6wvm krb5kdc[3768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266306, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:46.835747 4030 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:11:46.836549 4030 webserver.cc:460] Webserver started at http://127.1.253.66:40535/ using document root <none> and password file <none>
I20260924 16:11:46.837102 4030 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:46.837163 4030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:46.837324 4030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:46.839351 4030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "c9010c9ecf154abe86b2ab1834cde9f7"
format_stamp: "Formatted at 2026-09-24 16:11:46 on dist-test-slave-6wvm"
server_key: "b59f7061682e61d23b7c389d76e8e233"
server_key_iv: "3f5e1aa63eba00a0923c55f9d621d56a"
server_key_version: "encryptionkey@0"
I20260924 16:11:46.839854 4030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "c9010c9ecf154abe86b2ab1834cde9f7"
format_stamp: "Formatted at 2026-09-24 16:11:46 on dist-test-slave-6wvm"
server_key: "b59f7061682e61d23b7c389d76e8e233"
server_key_iv: "3f5e1aa63eba00a0923c55f9d621d56a"
server_key_version: "encryptionkey@0"
I20260924 16:11:46.843499 4030 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:11:46.845867 4045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:46.847198 4030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:11:46.847297 4030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "c9010c9ecf154abe86b2ab1834cde9f7"
format_stamp: "Formatted at 2026-09-24 16:11:46 on dist-test-slave-6wvm"
server_key: "b59f7061682e61d23b7c389d76e8e233"
server_key_iv: "3f5e1aa63eba00a0923c55f9d621d56a"
server_key_version: "encryptionkey@0"
I20260924 16:11:46.847383 4030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:46.881335 4030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:46.884618 4030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:46.884784 4030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:46.886221 4030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:46.886348 4030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:46.886421 4030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:46.886461 4030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:11:46 dist-test-slave-6wvm krb5kdc[3768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266306, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:46.897679 4030 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:33877
I20260924 16:11:46.898003 4164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:33877 every 8 connection(s)
I20260924 16:11:46.898793 4030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:11:46.900759 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:46.887312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:40347 (local address 127.1.253.126:42659)
0924 16:11:46.887503 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:46.887508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:46.888428 (+ 920us) server_negotiation.cc:407] Connection header received
0924 16:11:46.889772 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:46.889778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:46.889845 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:46.889978 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:46.891715 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.892354 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.893544 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.893779 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.893897 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:46.896732 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:46.896735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:46.896737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:46.896763 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:46.898539 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:46.899167 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:46.899172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:46.899175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:46.899276 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:46.899673 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:46.899677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:46.899678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:46.899937 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:46.900026 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:46.900504 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 16:11:46.900632 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:11:46.901592 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 4030
I20260924 16:11:46.901715 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:11:46.901791 4054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:46.887701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42659 (local address 127.1.253.66:40347)
0924 16:11:46.888226 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:46.888291 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:46.889456 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:46.890201 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:46.890213 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:46.890720 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:46.891515 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:46.891543 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.892488 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.892492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:46.893402 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:46.893411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.893952 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.893957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:46.894137 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:46.894803 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:46.894821 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:46.896539 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:46.898666 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:46.898672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:46.898687 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:46.899035 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:46.899394 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:46.899397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:46.899399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:46.899575 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:46.900036 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:46.900041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:46.900290 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 16:11:46.900450 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 16:11:46.900727 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260924 16:11:46.902022 2037 external_mini_cluster.cc:1428] Setting key 9fb55a4b42044bf8115612b75cc2c819
I20260924 16:11:46.903271 4165 heartbeater.cc:347] Connected to a master server at 127.1.253.126:42659
I20260924 16:11:46.903466 4165 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:46.904125 4165 heartbeater.cc:511] Master 127.1.253.126:42659 requested a full tablet report, sending...
I20260924 16:11:46.905364 3813 ts_manager.cc:194] Registered new tserver with Master: c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877)
I20260924 16:11:46.906170 3813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:40347
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:11:46.913925 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:46.906085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:52255 (local address 127.1.253.126:42659)
0924 16:11:46.906193 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:46.906198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:46.906212 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:11:46.906309 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:46.906312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:46.906353 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:46.906432 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:46.907007 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.907740 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.909119 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.909326 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.909410 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:46.910890 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:46.910893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:46.910895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:46.910917 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:46.912481 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:46.912865 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:46.912868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:46.912869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:46.912940 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:46.913218 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:46.913222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:46.913224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:46.913433 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:46.913514 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:46.913732 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 16:11:46.913818 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 16:11:46.913935 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:46.905775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42659 (local address 127.1.253.66:52255)
0924 16:11:46.906080 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:46.906098 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:46.906193 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:46.906471 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:46.906475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:46.906718 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:46.906872 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:46.906883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.907892 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.907899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:46.908984 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:46.908999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:46.909448 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:46.909454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:46.909599 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:46.910265 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:46.910281 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:46.910742 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:46.912597 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:46.912600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:46.912601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:46.912780 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:46.913032 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:46.913034 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:46.913035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:46.913105 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:46.913520 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:46.913523 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:46.913611 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 16:11:46.913695 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 16:11:46.913796 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":22,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:11:46.930609 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:42659
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:33791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:47.043861 4174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:47.044199 4174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:47.044281 4174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:47.048516 4174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:47.048617 4174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:47.048688 4174 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 16:11:47.048734 4174 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:11:47.048802 4174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:11:47.054469 4174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:33791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.253.126:42659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:47.055651 4174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:47.056751 4174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:47.066825 4182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:47.066857 4179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:47.066870 4180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:47.067059 4174 server_base.cc:1034] running on GCE node
I20260924 16:11:47.067644 4174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:47.068281 4174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:47.069442 4174 hybrid_clock.cc:648] HybridClock initialized: now 1790266307069424 us; error 28 us; skew 500 ppm
Sep 24 16:11:47 dist-test-slave-6wvm krb5kdc[3768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266307, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:47.072060 4174 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:11:47.072923 4174 webserver.cc:460] Webserver started at http://127.1.253.67:32809/ using document root <none> and password file <none>
I20260924 16:11:47.073510 4174 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:47.073596 4174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:47.073828 4174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:47.076071 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "be6d362797114bff9b6107b151235f49"
format_stamp: "Formatted at 2026-09-24 16:11:47 on dist-test-slave-6wvm"
server_key: "58b98e6fe5f94a8e0d2da1e9e2e98453"
server_key_iv: "2a4f44813a3cd602ab521ef1a8b67555"
server_key_version: "encryptionkey@0"
I20260924 16:11:47.076720 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "be6d362797114bff9b6107b151235f49"
format_stamp: "Formatted at 2026-09-24 16:11:47 on dist-test-slave-6wvm"
server_key: "58b98e6fe5f94a8e0d2da1e9e2e98453"
server_key_iv: "2a4f44813a3cd602ab521ef1a8b67555"
server_key_version: "encryptionkey@0"
I20260924 16:11:47.080390 4174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 16:11:47.082858 4189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:47.084007 4174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:11:47.084139 4174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "be6d362797114bff9b6107b151235f49"
format_stamp: "Formatted at 2026-09-24 16:11:47 on dist-test-slave-6wvm"
server_key: "58b98e6fe5f94a8e0d2da1e9e2e98453"
server_key_iv: "2a4f44813a3cd602ab521ef1a8b67555"
server_key_version: "encryptionkey@0"
I20260924 16:11:47.084249 4174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:47.113225 4174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:47.116426 4174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:47.116642 4174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:47.118199 4174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:47.118301 4174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:47.118383 4174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:47.118438 4174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:11:47 dist-test-slave-6wvm krb5kdc[3768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266307, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:47.129338 4174 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:45313
I20260924 16:11:47.129412 4306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:45313 every 8 connection(s)
I20260924 16:11:47.130493 4174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:11:47.132957 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.119227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:55917 (local address 127.1.253.126:42659)
0924 16:11:47.119420 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.119428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.120329 (+ 901us) server_negotiation.cc:407] Connection header received
0924 16:11:47.121484 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.121488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.121547 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.121651 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:47.123428 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.124038 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.125230 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.125601 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.125730 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.129183 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:47.129188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:47.129190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:47.129226 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:47.130779 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:47.131348 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:47.131351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:47.131353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:47.131466 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:47.131815 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:47.131819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:47.131821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:47.132007 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:47.132129 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.132676 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.132795 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:11:47.133747 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.119655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42659 (local address 127.1.253.67:55917)
0924 16:11:47.120148 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.120201 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.121219 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.121911 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.121924 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.122451 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:47.123236 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.123261 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.124178 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.124181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.125093 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.125103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.125790 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.125795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.126018 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.127270 (+ 1252us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:47.127286 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:47.128971 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:47.130893 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:47.130899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:47.130915 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:47.131232 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:47.131565 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:47.131568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:47.131570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:47.131716 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:47.132130 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:47.132137 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:47.132373 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.132635 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.132935 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":126,"threads_started":1}
I20260924 16:11:47.135178 4307 heartbeater.cc:347] Connected to a master server at 127.1.253.126:42659
I20260924 16:11:47.135367 4307 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:47.135912 4307 heartbeater.cc:511] Master 127.1.253.126:42659 requested a full tablet report, sending...
I20260924 16:11:47.137084 3812 ts_manager.cc:194] Registered new tserver with Master: be6d362797114bff9b6107b151235f49 (127.1.253.67:45313)
I20260924 16:11:47.137823 3812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:55917
I20260924 16:11:47.137863 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 4174
I20260924 16:11:47.138015 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:11:47.138429 2037 external_mini_cluster.cc:1428] Setting key 7293a445cfd360a427078bc3c8c3ae79
I20260924 16:11:47.140671 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:11:47.146410 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.137462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42659 (local address 127.1.253.67:41533)
0924 16:11:47.138430 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.138448 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.138612 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.138911 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.138917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.139137 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:47.139294 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.139304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.140322 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.140325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.141154 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.141167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.141658 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.141665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.141795 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.142361 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:47.142381 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:47.142836 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:47.144830 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:47.144833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:47.144835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:47.145027 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:47.145323 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:47.145325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:47.145326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:47.145399 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:47.145875 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:47.145880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:47.145985 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.146098 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.146198 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":94,"threads_started":1}
I20260924 16:11:47.146509 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.137555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:41533 (local address 127.1.253.126:42659)
0924 16:11:47.137700 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.137705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.138522 (+ 817us) server_negotiation.cc:407] Connection header received
0924 16:11:47.138725 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.138729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.138776 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.138862 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:47.139433 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.140185 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.141298 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.141532 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.141618 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.143000 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:47.143005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:47.143008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:47.143038 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:47.144709 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:47.145124 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:47.145131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:47.145135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:47.145228 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:47.145489 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:47.145496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:47.145500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:47.145754 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:47.145867 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.146197 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.146341 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 16:11:47 dist-test-slave-6wvm krb5kdc[3768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:11:47 dist-test-slave-6wvm krb5kdc[3768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:47.168727 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.160283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.1.253.126:42659)
0924 16:11:47.160514 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.160520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.160545 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:11:47.160589 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.160591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.160636 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.160719 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:47.161425 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.161948 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.162828 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.163064 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.163151 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.165581 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:47.165585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:47.165587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:47.165613 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:47.167319 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:47.167735 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:47.167738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:47.167739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:47.167821 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:47.168069 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:47.168071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:47.168072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:47.168244 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:47.168346 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.168512 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.168606 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:11:47.171691 3812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:11:47.174772 3812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:11:47.190207 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.183621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:33877 (local address 127.0.0.1:57610)
0924 16:11:47.184417 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.184482 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.184614 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.185295 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.185302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.185325 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:47.185472 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.185480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.187006 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.187009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.188465 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.188476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.189614 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.189618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.189736 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.189781 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.189904 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.190031 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":324,"thread_start_us":85,"threads_started":1}
I20260924 16:11:47.190799 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.183574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40767 (local address 127.0.0.1:42364)
0924 16:11:47.184814 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.184828 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.184904 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.185377 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.185379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.185398 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:47.185541 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.185550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.187007 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.187009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.188465 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.188476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.190356 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.190362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.190467 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.190494 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.190542 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.190615 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":107,"thread_start_us":62,"threads_started":1}
I20260924 16:11:47.190831 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.183931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:45313 (local address 127.0.0.1:56318)
0924 16:11:47.185529 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.185543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.185640 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.186024 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.186027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.186044 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:47.186126 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.186132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.187213 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.187216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.189201 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.189212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.190416 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.190420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.190500 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.190522 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.190594 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.190672 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1527,"mutex_wait_us":977,"thread_start_us":88,"threads_started":1}
I20260924 16:11:47.191047 4329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.184393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.1.253.66:33877)
0924 16:11:47.184847 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.184862 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.184887 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:11:47.184972 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.184976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.185139 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.185305 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:47.185609 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.186870 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.188609 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.189480 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.189585 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.189757 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.189953 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.190056 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":114,"threads_started":1}
I20260924 16:11:47.191294 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.184387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.1.253.67:45313)
0924 16:11:47.185145 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.185156 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.185631 (+ 475us) server_negotiation.cc:407] Connection header received
0924 16:11:47.185747 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.185752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.185888 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.186035 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:47.186220 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.187089 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.189319 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.190032 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.190148 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.190345 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.191035 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.191142 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":110,"threads_started":1}
I20260924 16:11:47.191637 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.183712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.1.253.65:40767)
0924 16:11:47.184906 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.184914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.184940 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:11:47.185021 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.185026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.185229 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.185396 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:47.185686 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.186869 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.189372 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.190189 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.191176 (+ 987us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.191332 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.191417 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.191508 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1062,"thread_start_us":102,"threads_started":1}
I20260924 16:11:47.194092 3952 tablet_service.cc:1467] Processing CreateTablet for tablet 2063a76651fd4494a52ae68c772b9b12 (DEFAULT_TABLE table=test-table [id=fbb772a016e145958530be41debd0755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:47.194180 4239 tablet_service.cc:1467] Processing CreateTablet for tablet 2063a76651fd4494a52ae68c772b9b12 (DEFAULT_TABLE table=test-table [id=fbb772a016e145958530be41debd0755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:47.194180 4095 tablet_service.cc:1467] Processing CreateTablet for tablet 2063a76651fd4494a52ae68c772b9b12 (DEFAULT_TABLE table=test-table [id=fbb772a016e145958530be41debd0755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:47.195286 3952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2063a76651fd4494a52ae68c772b9b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:47.195289 4095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2063a76651fd4494a52ae68c772b9b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:47.195293 4239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2063a76651fd4494a52ae68c772b9b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:47.201473 4330 tablet_bootstrap.cc:492] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6: Bootstrap starting.
I20260924 16:11:47.203609 4331 tablet_bootstrap.cc:492] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49: Bootstrap starting.
I20260924 16:11:47.203990 4332 tablet_bootstrap.cc:492] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7: Bootstrap starting.
I20260924 16:11:47.204166 4330 tablet_bootstrap.cc:654] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:47.205678 4330 log.cc:824] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:47.206039 4331 tablet_bootstrap.cc:654] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:47.206288 4332 tablet_bootstrap.cc:654] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:47.206998 4331 log.cc:824] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:47.207012 4332 log.cc:824] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:47.207791 4330 tablet_bootstrap.cc:492] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6: No bootstrap required, opened a new log
I20260924 16:11:47.207949 4330 ts_tablet_manager.cc:1397] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 16:11:47.211776 4330 raft_consensus.cc:348] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.212105 4331 tablet_bootstrap.cc:492] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49: No bootstrap required, opened a new log
I20260924 16:11:47.212127 4330 raft_consensus.cc:374] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:47.212229 4330 raft_consensus.cc:729] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0603fc7358f84df0bef56cf51fb760b6, State: Initialized, Role: FOLLOWER
I20260924 16:11:47.212345 4331 ts_tablet_manager.cc:1397] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260924 16:11:47.212652 4332 tablet_bootstrap.cc:492] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7: No bootstrap required, opened a new log
I20260924 16:11:47.212776 4330 consensus_queue.cc:260] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.212880 4332 ts_tablet_manager.cc:1397] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260924 16:11:47.213469 4020 heartbeater.cc:503] Master 127.1.253.126:42659 was elected leader, sending a full tablet report...
I20260924 16:11:47.213783 4330 ts_tablet_manager.cc:1428] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:11:47.216609 4332 raft_consensus.cc:348] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.216967 4332 raft_consensus.cc:374] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:47.217060 4332 raft_consensus.cc:729] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9010c9ecf154abe86b2ab1834cde9f7, State: Initialized, Role: FOLLOWER
I20260924 16:11:47.216881 4331 raft_consensus.cc:348] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.217154 4331 raft_consensus.cc:374] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:47.217216 4331 raft_consensus.cc:729] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6d362797114bff9b6107b151235f49, State: Initialized, Role: FOLLOWER
I20260924 16:11:47.217576 4332 consensus_queue.cc:260] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.217671 4331 consensus_queue.cc:260] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.218416 4165 heartbeater.cc:503] Master 127.1.253.126:42659 was elected leader, sending a full tablet report...
I20260924 16:11:47.218775 4331 ts_tablet_manager.cc:1428] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260924 16:11:47.218829 4332 ts_tablet_manager.cc:1428] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:11:47.219110 4307 heartbeater.cc:503] Master 127.1.253.126:42659 was elected leader, sending a full tablet report...
W20260924 16:11:47.381548 4308 tablet.cc:2369] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:11:47.400133 4166 tablet.cc:2369] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:11:47.410372 4021 tablet.cc:2369] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:11:47.536764 4336 raft_consensus.cc:482] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:47.537050 4336 raft_consensus.cc:504] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.538478 4336 leader_election.cc:288] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877), be6d362797114bff9b6107b151235f49 (127.1.253.67:45313)
I20260924 16:11:47.542537 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.539045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:56487 (local address 127.1.253.67:45313)
0924 16:11:47.539209 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.539213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.539230 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:11:47.539322 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.539325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.539405 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.539498 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:47.539836 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.540376 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.541525 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.541921 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.542052 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.542107 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.542328 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.542406 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:11:47.542567 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.538780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:45313 (local address 127.1.253.65:56487)
0924 16:11:47.539103 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.539118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.539225 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.539511 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.539514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.539553 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:47.539717 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.539722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.540496 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.540501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.541393 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.541402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.542046 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.542049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.542146 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.542167 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.542306 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.542378 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":51,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1}
I20260924 16:11:47.542599 4329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.538781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:51141 (local address 127.1.253.66:33877)
0924 16:11:47.538943 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.538949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.539155 (+ 206us) server_negotiation.cc:407] Connection header received
0924 16:11:47.539314 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.539317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.539371 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.539467 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:47.539839 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.540377 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.541551 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.541954 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.542095 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.542158 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.542380 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.542463 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:11:47.543092 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.538662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:33877 (local address 127.1.253.65:51141)
0924 16:11:47.539006 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.539024 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.539169 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.539481 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.539484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.539502 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:47.539711 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.539717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.540504 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.540510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.541410 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.541420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.542086 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.542091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.542204 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.542232 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.542353 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.542433 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":85,"spinlock_wait_cycles":10496,"thread_start_us":127,"threads_started":1}
I20260924 16:11:47.543329 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2063a76651fd4494a52ae68c772b9b12" candidate_uuid: "0603fc7358f84df0bef56cf51fb760b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be6d362797114bff9b6107b151235f49" is_pre_election: true
I20260924 16:11:47.543695 4259 raft_consensus.cc:2454] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0603fc7358f84df0bef56cf51fb760b6 in term 0.
I20260924 16:11:47.543897 4116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2063a76651fd4494a52ae68c772b9b12" candidate_uuid: "0603fc7358f84df0bef56cf51fb760b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" is_pre_election: true
I20260924 16:11:47.544174 3905 leader_election.cc:302] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0603fc7358f84df0bef56cf51fb760b6, be6d362797114bff9b6107b151235f49; no voters:
I20260924 16:11:47.544224 4116 raft_consensus.cc:2454] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0603fc7358f84df0bef56cf51fb760b6 in term 0.
I20260924 16:11:47.544477 4336 raft_consensus.cc:2781] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:47.544564 4336 raft_consensus.cc:482] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:47.544610 4336 raft_consensus.cc:3037] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:47.546152 4336 raft_consensus.cc:504] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.546535 4336 leader_election.cc:288] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [CANDIDATE]: Term 1 election: Requested vote from peers c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877), be6d362797114bff9b6107b151235f49 (127.1.253.67:45313)
I20260924 16:11:47.546886 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2063a76651fd4494a52ae68c772b9b12" candidate_uuid: "0603fc7358f84df0bef56cf51fb760b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be6d362797114bff9b6107b151235f49"
I20260924 16:11:47.546921 4116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2063a76651fd4494a52ae68c772b9b12" candidate_uuid: "0603fc7358f84df0bef56cf51fb760b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9010c9ecf154abe86b2ab1834cde9f7"
I20260924 16:11:47.547037 4259 raft_consensus.cc:3037] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:47.547083 4116 raft_consensus.cc:3037] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:47.548161 4116 raft_consensus.cc:2454] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0603fc7358f84df0bef56cf51fb760b6 in term 1.
I20260924 16:11:47.548322 4259 raft_consensus.cc:2454] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0603fc7358f84df0bef56cf51fb760b6 in term 1.
I20260924 16:11:47.548533 3904 leader_election.cc:302] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0603fc7358f84df0bef56cf51fb760b6, c9010c9ecf154abe86b2ab1834cde9f7; no voters:
I20260924 16:11:47.548730 4336 raft_consensus.cc:2781] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:47.548971 4336 raft_consensus.cc:686] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0603fc7358f84df0bef56cf51fb760b6, State: Running, Role: LEADER
I20260924 16:11:47.549340 4336 consensus_queue.cc:237] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.552866 3813 catalog_manager.cc:5495] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65). New cstate: current_term: 1 leader_uuid: "0603fc7358f84df0bef56cf51fb760b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } health_report { overall_health: UNKNOWN } } }
Sep 24 16:11:47 dist-test-slave-6wvm krb5kdc[3768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266306, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:47.665386 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.656112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.1.253.126:42659)
0924 16:11:47.656261 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.656269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.656490 (+ 221us) server_negotiation.cc:407] Connection header received
0924 16:11:47.656653 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.656658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.656702 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.656790 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:47.657460 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.658045 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.659033 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.659258 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.659418 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.661844 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:47.661851 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:47.661857 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:47.661897 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:47.663529 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:47.664057 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:47.664062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:47.664064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:47.664146 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:47.664451 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:47.664456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:47.664461 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:47.664698 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:47.664803 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.665106 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.665251 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":53}
I20260924 16:11:47.665400 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.656085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42659 (local address 127.0.0.1:42788)
0924 16:11:47.656398 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.656410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.656554 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.656808 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.656810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.657244 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:47.657341 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.657348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.658194 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.658199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.658902 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.658912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.659418 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.659422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.659496 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.660124 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:47.660136 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:47.661692 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:47.663646 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:47.663654 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:47.663659 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:47.663908 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:47.664259 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:47.664262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:47.664263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:47.664335 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:47.664802 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:47.664809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:47.664920 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.665036 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.665197 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":124,"threads_started":1}
I20260924 16:11:47.669386 3813 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:42788:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:11:47.670042 3813 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:11:47.674995 4095 tablet_service.cc:1467] Processing CreateTablet for tablet acaa59edd4f7454591abbd0ce23557dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8981ddb788a46f69e29aa9389b27b04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:11:47.674795 3952 tablet_service.cc:1467] Processing CreateTablet for tablet acaa59edd4f7454591abbd0ce23557dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8981ddb788a46f69e29aa9389b27b04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:11:47.675104 4239 tablet_service.cc:1467] Processing CreateTablet for tablet acaa59edd4f7454591abbd0ce23557dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8981ddb788a46f69e29aa9389b27b04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:11:47.675583 3952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acaa59edd4f7454591abbd0ce23557dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:47.675602 4095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acaa59edd4f7454591abbd0ce23557dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:47.675640 4239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acaa59edd4f7454591abbd0ce23557dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:47.678119 4332 tablet_bootstrap.cc:492] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7: Bootstrap starting.
I20260924 16:11:47.678453 4330 tablet_bootstrap.cc:492] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6: Bootstrap starting.
I20260924 16:11:47.678503 4331 tablet_bootstrap.cc:492] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: Bootstrap starting.
I20260924 16:11:47.679275 4332 tablet_bootstrap.cc:654] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:47.679415 4330 tablet_bootstrap.cc:654] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:47.679494 4331 tablet_bootstrap.cc:654] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:47.680735 4331 tablet_bootstrap.cc:492] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: No bootstrap required, opened a new log
I20260924 16:11:47.680846 4331 ts_tablet_manager.cc:1397] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:11:47.680872 4332 tablet_bootstrap.cc:492] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7: No bootstrap required, opened a new log
I20260924 16:11:47.680959 4332 ts_tablet_manager.cc:1397] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:11:47.681241 4330 tablet_bootstrap.cc:492] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6: No bootstrap required, opened a new log
I20260924 16:11:47.681326 4330 ts_tablet_manager.cc:1397] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:11:47.681381 4331 raft_consensus.cc:348] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.681421 4332 raft_consensus.cc:348] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.681514 4331 raft_consensus.cc:374] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:47.681529 4332 raft_consensus.cc:374] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:47.681557 4331 raft_consensus.cc:729] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6d362797114bff9b6107b151235f49, State: Initialized, Role: FOLLOWER
I20260924 16:11:47.681565 4332 raft_consensus.cc:729] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9010c9ecf154abe86b2ab1834cde9f7, State: Initialized, Role: FOLLOWER
I20260924 16:11:47.681677 4331 consensus_queue.cc:260] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.681702 4332 consensus_queue.cc:260] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.681806 4330 raft_consensus.cc:348] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.681915 4330 raft_consensus.cc:374] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:47.681948 4330 raft_consensus.cc:729] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0603fc7358f84df0bef56cf51fb760b6, State: Initialized, Role: FOLLOWER
I20260924 16:11:47.681960 4332 ts_tablet_manager.cc:1428] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 16:11:47.681980 4331 ts_tablet_manager.cc:1428] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:11:47.682083 4330 consensus_queue.cc:260] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.682399 4338 tablet_replica.cc:439] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } } }
I20260924 16:11:47.682556 4338 tablet_replica.cc:442] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:47.682778 4337 tablet_replica.cc:439] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } } }
I20260924 16:11:47.682924 4337 tablet_replica.cc:442] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:47.683983 4336 tablet_replica.cc:439] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } } }
I20260924 16:11:47.684120 4336 tablet_replica.cc:442] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:47.684340 4330 ts_tablet_manager.cc:1428] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 16:11:47.860525 4338 raft_consensus.cc:482] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:47.860688 4338 raft_consensus.cc:504] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.862089 4338 leader_election.cc:288] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767), c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877)
I20260924 16:11:47.866935 4329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.862755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:40711 (local address 127.1.253.66:33877)
0924 16:11:47.862891 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.862898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.862919 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:11:47.863065 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.863067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.863109 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.863205 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:47.863562 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.864174 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.865868 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.866322 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.866456 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.866515 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.866728 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.866799 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:11:47.866958 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.862503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:33877 (local address 127.1.253.67:40711)
0924 16:11:47.862832 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.862848 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.862954 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.863237 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.863240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.863274 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:47.863431 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.863441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.864279 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.864282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.865744 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.865752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.866465 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.866469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.866584 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.866616 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.866709 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.866792 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":130,"thread_start_us":88,"threads_started":1}
I20260924 16:11:47.867430 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.862351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40767 (local address 127.1.253.67:41873)
0924 16:11:47.862703 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.862722 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.862858 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.863210 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.863213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.863231 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:47.863376 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.863383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.864210 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.864215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.865740 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.865752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.866973 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.866977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.867085 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.867114 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.867205 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.867278 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":102,"spinlock_wait_cycles":9600,"thread_start_us":127,"threads_started":1}
I20260924 16:11:47.867443 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.862490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:41873 (local address 127.1.253.65:40767)
0924 16:11:47.862923 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.862928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.862966 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:11:47.863028 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.863032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.863094 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.863164 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:47.863505 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.864089 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.866421 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.866844 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.866985 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.867038 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.867227 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.867310 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":78,"threads_started":1}
I20260924 16:11:47.867885 4116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaa59edd4f7454591abbd0ce23557dc" candidate_uuid: "be6d362797114bff9b6107b151235f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" is_pre_election: true
I20260924 16:11:47.868065 4116 raft_consensus.cc:2454] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be6d362797114bff9b6107b151235f49 in term 0.
I20260924 16:11:47.868155 3972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaa59edd4f7454591abbd0ce23557dc" candidate_uuid: "be6d362797114bff9b6107b151235f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0603fc7358f84df0bef56cf51fb760b6" is_pre_election: true
I20260924 16:11:47.868414 3972 raft_consensus.cc:2454] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be6d362797114bff9b6107b151235f49 in term 0.
I20260924 16:11:47.868454 4191 leader_election.cc:302] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be6d362797114bff9b6107b151235f49, c9010c9ecf154abe86b2ab1834cde9f7; no voters:
I20260924 16:11:47.868805 4338 raft_consensus.cc:2781] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:47.868884 4338 raft_consensus.cc:482] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:47.868916 4338 raft_consensus.cc:3037] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:47.869959 4338 raft_consensus.cc:504] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.870352 4338 leader_election.cc:288] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [CANDIDATE]: Term 1 election: Requested vote from peers 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767), c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877)
I20260924 16:11:47.870735 3972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaa59edd4f7454591abbd0ce23557dc" candidate_uuid: "be6d362797114bff9b6107b151235f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0603fc7358f84df0bef56cf51fb760b6"
I20260924 16:11:47.870752 4116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaa59edd4f7454591abbd0ce23557dc" candidate_uuid: "be6d362797114bff9b6107b151235f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9010c9ecf154abe86b2ab1834cde9f7"
I20260924 16:11:47.870858 4116 raft_consensus.cc:3037] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:47.870858 3972 raft_consensus.cc:3037] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:47.872026 3972 raft_consensus.cc:2454] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be6d362797114bff9b6107b151235f49 in term 1.
I20260924 16:11:47.872026 4116 raft_consensus.cc:2454] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be6d362797114bff9b6107b151235f49 in term 1.
I20260924 16:11:47.872355 4190 leader_election.cc:302] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0603fc7358f84df0bef56cf51fb760b6, be6d362797114bff9b6107b151235f49; no voters:
I20260924 16:11:47.872532 4338 raft_consensus.cc:2781] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:47.872706 4338 raft_consensus.cc:686] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [term 1 LEADER]: Becoming Leader. State: Replica: be6d362797114bff9b6107b151235f49, State: Running, Role: LEADER
I20260924 16:11:47.873047 4338 consensus_queue.cc:237] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:47.875612 4355 tablet_replica.cc:439] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: TxnStatusTablet state changed. Reason: New leader be6d362797114bff9b6107b151235f49. Latest consensus state: current_term: 1 leader_uuid: "be6d362797114bff9b6107b151235f49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } } }
I20260924 16:11:47.875723 4355 tablet_replica.cc:442] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:11:47.876049 4357 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:11:47.876408 4357 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:11:47.876797 3813 catalog_manager.cc:5495] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 reported cstate change: term changed from 0 to 1, leader changed from <none> to be6d362797114bff9b6107b151235f49 (127.1.253.67). New cstate: current_term: 1 leader_uuid: "be6d362797114bff9b6107b151235f49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } health_report { overall_health: HEALTHY } } }
I20260924 16:11:47.924827 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.920782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:45313 (local address 127.0.0.1:56332)
0924 16:11:47.920963 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.920983 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.921102 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.921505 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.921509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.921517 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:11:47.921640 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.921646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.922462 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.922468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.923082 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.923110 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.923603 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.923608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.923707 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.923753 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:11:47.924490 (+ 737us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:11:47.924508 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.924587 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.924680 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 16:11:47.924827 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.920913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.1.253.67:45313)
0924 16:11:47.921061 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.921066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.921083 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:11:47.921289 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.921295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.921372 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.921486 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:47.921762 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.922331 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.923236 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.923478 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.923601 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.923932 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:47.924379 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:47.924518 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.924597 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.924650 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:11:47.930222 3972 raft_consensus.cc:1261] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Refusing update from remote peer be6d362797114bff9b6107b151235f49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:47.930331 4116 raft_consensus.cc:1261] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Refusing update from remote peer be6d362797114bff9b6107b151235f49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:47.931046 4338 consensus_queue.cc:1035] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:47.931254 4355 consensus_queue.cc:1035] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:47.934635 4336 tablet_replica.cc:439] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6: TxnStatusTablet state changed. Reason: New leader be6d362797114bff9b6107b151235f49. Latest consensus state: current_term: 1 leader_uuid: "be6d362797114bff9b6107b151235f49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } } }
I20260924 16:11:47.934794 4336 tablet_replica.cc:442] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:47.935753 4337 tablet_replica.cc:439] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7: TxnStatusTablet state changed. Reason: New leader be6d362797114bff9b6107b151235f49. Latest consensus state: current_term: 1 leader_uuid: "be6d362797114bff9b6107b151235f49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } } }
I20260924 16:11:47.935868 4337 tablet_replica.cc:442] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:47.939443 4338 tablet_replica.cc:439] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be6d362797114bff9b6107b151235f49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } } }
I20260924 16:11:47.939574 4338 tablet_replica.cc:442] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:11:47.940448 4355 tablet_replica.cc:439] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be6d362797114bff9b6107b151235f49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } } }
I20260924 16:11:47.940550 4355 tablet_replica.cc:442] T acaa59edd4f7454591abbd0ce23557dc P be6d362797114bff9b6107b151235f49: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:11:47.941656 4337 tablet_replica.cc:439] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be6d362797114bff9b6107b151235f49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } } }
I20260924 16:11:47.941757 4337 tablet_replica.cc:442] T acaa59edd4f7454591abbd0ce23557dc P c9010c9ecf154abe86b2ab1834cde9f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:47.942356 4362 mvcc.cc:205] Tried to move back new op lower bound from 7332930797274841088 to 7332930797050929152. Current Snapshot: MvccSnapshot[applied={T|T < 7332930797274841088}]
I20260924 16:11:47.944176 4361 mvcc.cc:205] Tried to move back new op lower bound from 7332930797274841088 to 7332930797050929152. Current Snapshot: MvccSnapshot[applied={T|T < 7332930797274841088}]
I20260924 16:11:47.945853 4336 tablet_replica.cc:439] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be6d362797114bff9b6107b151235f49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } } }
I20260924 16:11:47.945986 4336 tablet_replica.cc:442] T acaa59edd4f7454591abbd0ce23557dc P 0603fc7358f84df0bef56cf51fb760b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:11:47.950309 4336 consensus_queue.cc:1035] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:47.951049 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.946744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.1.253.65:40767)
0924 16:11:47.946889 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.946896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.946914 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:11:47.946995 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.946998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.947064 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.947183 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:47.947553 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.948172 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.949235 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.949579 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.949690 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.950036 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:47.950591 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:47.950711 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.950842 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.950908 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:11:47.955971 4341 consensus_queue.cc:1035] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:47.965677 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.961584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:45313 (local address 127.1.253.65:40993)
0924 16:11:47.961921 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.961933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.962022 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.962294 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.962297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.962305 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:11:47.962427 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.962431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.963481 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.963484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.964252 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.964262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.964920 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.964925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.965003 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.965053 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:11:47.965393 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:11:47.965399 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.965458 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.965533 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":85,"threads_started":1}
I20260924 16:11:47.965682 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.961645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:40993 (local address 127.1.253.67:45313)
0924 16:11:47.961777 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.961782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.962000 (+ 218us) server_negotiation.cc:407] Connection header received
0924 16:11:47.962102 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.962104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.962155 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.962255 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:47.962526 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.963369 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.964396 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.964735 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.964839 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.965216 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:47.965300 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:47.965372 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.965490 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.965549 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:11:47.997090 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.992536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:40689 (local address 127.1.253.65:40767)
0924 16:11:47.992686 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:47.992691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:47.992939 (+ 248us) server_negotiation.cc:407] Connection header received
0924 16:11:47.993050 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:47.993055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:47.993140 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:47.993225 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:47.993619 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.994451 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.995776 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.996005 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.996096 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.996499 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:47.996623 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:47.996709 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:47.996865 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 16:11:47.996927 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:11:47.997767 4378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:47.992476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40767 (local address 127.1.253.67:40689)
0924 16:11:47.992828 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:47.992843 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:47.992949 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:47.993342 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:47.993345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:47.993354 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:11:47.993494 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.993501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.994606 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.994610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.995625 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:47.995639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:47.996167 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:47.996171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:47.996271 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:47.996331 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:11:47.996735 (+ 404us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:11:47.996743 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 16:11:47.997247 (+ 504us) client_negotiation.cc:240] Negotiation successful
0924 16:11:47.997337 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":121,"threads_started":1}
I20260924 16:11:48.025465 3812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:48.025676 3812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:48.028743 3812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66: Sending DeleteTablet for 3 replicas of tablet 2063a76651fd4494a52ae68c772b9b12
I20260924 16:11:48.029604 3952 tablet_service.cc:1514] Processing DeleteTablet for tablet 2063a76651fd4494a52ae68c772b9b12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:48 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:42364
I20260924 16:11:48.029785 4095 tablet_service.cc:1514] Processing DeleteTablet for tablet 2063a76651fd4494a52ae68c772b9b12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:48 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:57610
I20260924 16:11:48.030637 4239 tablet_service.cc:1514] Processing DeleteTablet for tablet 2063a76651fd4494a52ae68c772b9b12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:48 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:56318
I20260924 16:11:48.030735 4383 tablet_replica.cc:330] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7: stopping tablet replica
I20260924 16:11:48.031098 4383 raft_consensus.cc:2229] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:48.031338 4383 raft_consensus.cc:2258] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:48.031754 4382 tablet_replica.cc:330] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6: stopping tablet replica
I20260924 16:11:48.031978 3812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:11:48.032124 4384 tablet_replica.cc:330] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49: stopping tablet replica
W20260924 16:11:48.032461 3812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:11:48.032729 4384 raft_consensus.cc:2229] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:48.032948 4384 raft_consensus.cc:2258] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:48.034379 4384 ts_tablet_manager.cc:1905] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:48.037608 4384 ts_tablet_manager.cc:1918] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:11:48.037771 4384 log.cc:1196] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/2063a76651fd4494a52ae68c772b9b12
I20260924 16:11:48.038182 4384 ts_tablet_manager.cc:1939] T 2063a76651fd4494a52ae68c772b9b12 P be6d362797114bff9b6107b151235f49: Deleting consensus metadata
I20260924 16:11:48.039254 4095 tablet_service.cc:1467] Processing CreateTablet for tablet 97f156ade85a4735902c59b014140e76 (DEFAULT_TABLE table=test-table [id=26c8e6559f61479181b48d3a0827d893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:48.039573 4095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f156ade85a4735902c59b014140e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:48.039606 3799 catalog_manager.cc:4841] TS be6d362797114bff9b6107b151235f49 (127.1.253.67:45313): tablet 2063a76651fd4494a52ae68c772b9b12 (table test-table [id=fbb772a016e145958530be41debd0755]) successfully deleted
I20260924 16:11:48.042037 4332 tablet_bootstrap.cc:492] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7: Bootstrap starting.
I20260924 16:11:48.042208 4239 tablet_service.cc:1467] Processing CreateTablet for tablet 97f156ade85a4735902c59b014140e76 (DEFAULT_TABLE table=test-table [id=26c8e6559f61479181b48d3a0827d893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:48.042487 4239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f156ade85a4735902c59b014140e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:48.043044 4332 tablet_bootstrap.cc:654] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:48.044449 4332 tablet_bootstrap.cc:492] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7: No bootstrap required, opened a new log
I20260924 16:11:48.044539 4332 ts_tablet_manager.cc:1397] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:11:48.044715 4331 tablet_bootstrap.cc:492] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49: Bootstrap starting.
I20260924 16:11:48.044967 4332 raft_consensus.cc:348] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:48.045085 4332 raft_consensus.cc:374] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:48.045117 4332 raft_consensus.cc:729] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9010c9ecf154abe86b2ab1834cde9f7, State: Initialized, Role: FOLLOWER
I20260924 16:11:48.045243 4332 consensus_queue.cc:260] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:48.045478 4332 ts_tablet_manager.cc:1428] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:11:48.045523 4331 tablet_bootstrap.cc:654] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:48.048230 4383 ts_tablet_manager.cc:1905] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:48.050990 4331 tablet_bootstrap.cc:492] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49: No bootstrap required, opened a new log
I20260924 16:11:48.051071 4331 ts_tablet_manager.cc:1397] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260924 16:11:48.051153 4383 ts_tablet_manager.cc:1918] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:11:48.051239 4383 log.cc:1196] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/2063a76651fd4494a52ae68c772b9b12
I20260924 16:11:48.051549 4383 ts_tablet_manager.cc:1939] T 2063a76651fd4494a52ae68c772b9b12 P c9010c9ecf154abe86b2ab1834cde9f7: Deleting consensus metadata
I20260924 16:11:48.051499 4331 raft_consensus.cc:348] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:48.051604 4331 raft_consensus.cc:374] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:48.051631 4331 raft_consensus.cc:729] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6d362797114bff9b6107b151235f49, State: Initialized, Role: FOLLOWER
I20260924 16:11:48.051745 4331 consensus_queue.cc:260] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:48.051982 4331 ts_tablet_manager.cc:1428] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 16:11:48.052783 3798 catalog_manager.cc:4841] TS c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877): tablet 2063a76651fd4494a52ae68c772b9b12 (table test-table [id=fbb772a016e145958530be41debd0755]) successfully deleted
I20260924 16:11:48.046916 3952 tablet_service.cc:1467] Processing CreateTablet for tablet 97f156ade85a4735902c59b014140e76 (DEFAULT_TABLE table=test-table [id=26c8e6559f61479181b48d3a0827d893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:48.053215 3952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f156ade85a4735902c59b014140e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:48.055668 4330 tablet_bootstrap.cc:492] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6: Bootstrap starting.
I20260924 16:11:48.056597 4330 tablet_bootstrap.cc:654] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:48.057881 4330 tablet_bootstrap.cc:492] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6: No bootstrap required, opened a new log
I20260924 16:11:48.057967 4330 ts_tablet_manager.cc:1397] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:11:48.058482 4330 raft_consensus.cc:348] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:48.058596 4330 raft_consensus.cc:374] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:48.058626 4330 raft_consensus.cc:729] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0603fc7358f84df0bef56cf51fb760b6, State: Initialized, Role: FOLLOWER
I20260924 16:11:48.058750 4330 consensus_queue.cc:260] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:48.059026 4330 ts_tablet_manager.cc:1428] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 16:11:48.060024 4382 raft_consensus.cc:2229] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:11:48.060436 4382 raft_consensus.cc:2258] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:48.062189 4382 ts_tablet_manager.cc:1905] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:48.065220 4382 ts_tablet_manager.cc:1918] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:11:48.065318 4382 log.cc:1196] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/wal/wals/2063a76651fd4494a52ae68c772b9b12
I20260924 16:11:48.065639 4382 ts_tablet_manager.cc:1939] T 2063a76651fd4494a52ae68c772b9b12 P 0603fc7358f84df0bef56cf51fb760b6: Deleting consensus metadata
I20260924 16:11:48.066867 3797 catalog_manager.cc:4841] TS 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767): tablet 2063a76651fd4494a52ae68c772b9b12 (table test-table [id=fbb772a016e145958530be41debd0755]) successfully deleted
I20260924 16:11:48.453434 4337 raft_consensus.cc:482] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:48.453595 4337 raft_consensus.cc:504] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:48.454730 4337 leader_election.cc:288] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767), be6d362797114bff9b6107b151235f49 (127.1.253.67:45313)
I20260924 16:11:48.458935 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:48.455465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:58477 (local address 127.1.253.67:45313)
0924 16:11:48.455629 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:48.455635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:48.455651 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:11:48.455710 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:48.455714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:48.455777 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:48.455845 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:48.456182 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.456705 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.457873 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.458356 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.458451 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:48.458552 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:48.458714 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 16:11:48.458797 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:11:48.458977 4388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:48.455064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:45313 (local address 127.1.253.66:58477)
0924 16:11:48.455339 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:48.455357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:48.455479 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:48.455898 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:48.455903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:48.455923 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:48.456065 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:48.456071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.456818 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.456824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:48.457721 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:48.457730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.458474 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.458481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:48.458577 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:48.458610 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:11:48.458683 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 16:11:48.458757 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260924 16:11:48.459065 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:48.455179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:35273 (local address 127.1.253.65:40767)
0924 16:11:48.455335 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:48.455340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:48.455650 (+ 310us) server_negotiation.cc:407] Connection header received
0924 16:11:48.455777 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:48.455780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:48.455846 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:48.455930 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:11:48.456228 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.456730 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.457837 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.458395 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.458529 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:48.458584 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:48.458852 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 16:11:48.458929 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:11:48.459254 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:48.455065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40767 (local address 127.1.253.66:35273)
0924 16:11:48.455496 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:48.455532 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:48.455676 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:48.455961 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:48.455964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:48.455988 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:11:48.456088 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:48.456093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.456868 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.456871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:48.457712 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:48.457721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.458569 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.458574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:48.458717 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:48.458742 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:11:48.458819 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 16:11:48.458885 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":127,"thread_start_us":88,"threads_started":1}
I20260924 16:11:48.459712 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f156ade85a4735902c59b014140e76" candidate_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be6d362797114bff9b6107b151235f49" is_pre_election: true
I20260924 16:11:48.459712 3972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f156ade85a4735902c59b014140e76" candidate_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0603fc7358f84df0bef56cf51fb760b6" is_pre_election: true
I20260924 16:11:48.459872 4259 raft_consensus.cc:2454] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9010c9ecf154abe86b2ab1834cde9f7 in term 0.
I20260924 16:11:48.459874 3972 raft_consensus.cc:2454] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9010c9ecf154abe86b2ab1834cde9f7 in term 0.
I20260924 16:11:48.460255 4048 leader_election.cc:302] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0603fc7358f84df0bef56cf51fb760b6, be6d362797114bff9b6107b151235f49, c9010c9ecf154abe86b2ab1834cde9f7; no voters:
I20260924 16:11:48.460553 4337 raft_consensus.cc:2781] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:48.460620 4337 raft_consensus.cc:482] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:48.460649 4337 raft_consensus.cc:3037] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:48.461686 4337 raft_consensus.cc:504] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:48.462095 4337 leader_election.cc:288] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [CANDIDATE]: Term 1 election: Requested vote from peers 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767), be6d362797114bff9b6107b151235f49 (127.1.253.67:45313)
I20260924 16:11:48.462437 3972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f156ade85a4735902c59b014140e76" candidate_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0603fc7358f84df0bef56cf51fb760b6"
I20260924 16:11:48.462570 3972 raft_consensus.cc:3037] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:48.462533 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f156ade85a4735902c59b014140e76" candidate_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be6d362797114bff9b6107b151235f49"
I20260924 16:11:48.462668 4259 raft_consensus.cc:3037] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:48.463604 3972 raft_consensus.cc:2454] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9010c9ecf154abe86b2ab1834cde9f7 in term 1.
I20260924 16:11:48.463667 4259 raft_consensus.cc:2454] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9010c9ecf154abe86b2ab1834cde9f7 in term 1.
I20260924 16:11:48.463904 4046 leader_election.cc:302] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0603fc7358f84df0bef56cf51fb760b6, c9010c9ecf154abe86b2ab1834cde9f7; no voters:
I20260924 16:11:48.464087 4337 raft_consensus.cc:2781] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:48.464301 4337 raft_consensus.cc:686] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 LEADER]: Becoming Leader. State: Replica: c9010c9ecf154abe86b2ab1834cde9f7, State: Running, Role: LEADER
I20260924 16:11:48.464561 4337 consensus_queue.cc:237] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:48.467104 3813 catalog_manager.cc:5495] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66). New cstate: current_term: 1 leader_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } health_report { overall_health: UNKNOWN } } }
I20260924 16:11:48.517271 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:48.513213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.1.253.66:33877)
0924 16:11:48.513482 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:48.513488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:48.513501 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:11:48.513599 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:48.513601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:48.513654 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:48.513763 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:48.514019 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.514688 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.515493 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.515727 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.515805 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:48.516284 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:48.516812 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:48.516925 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:48.517057 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 16:11:48.517107 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":118,"threads_started":1}
I20260924 16:11:48.530023 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:48.527098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:59943 (local address 127.1.253.67:45313)
0924 16:11:48.527255 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:48.527258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:48.527307 (+ 49us) server_negotiation.cc:407] Connection header received
0924 16:11:48.527456 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:48.527459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:48.527506 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:48.527583 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:48.527799 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.528297 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.529108 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.529322 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.529403 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:48.529649 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:48.529718 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:48.529774 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:48.529852 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 16:11:48.529914 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:11:48.530038 4398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:48.526966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:45313 (local address 127.1.253.66:59943)
0924 16:11:48.527228 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:48.527241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:48.527336 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:48.527597 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:48.527599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:48.527605 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:11:48.527711 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:48.527716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.528382 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.528384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:48.529002 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:48.529009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.529418 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.529421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:48.529487 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:48.529529 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:11:48.529791 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:11:48.529796 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:11:48.529843 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 16:11:48.529913 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260924 16:11:48.536156 4259 raft_consensus.cc:1261] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Refusing update from remote peer c9010c9ecf154abe86b2ab1834cde9f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:48.536651 4390 consensus_queue.cc:1035] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:48.536775 3972 raft_consensus.cc:1261] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Refusing update from remote peer c9010c9ecf154abe86b2ab1834cde9f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:48.537223 4390 consensus_queue.cc:1035] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:48.554255 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:48.550825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:56151 (local address 127.1.253.66:33877)
0924 16:11:48.550932 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:48.550937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:48.551100 (+ 163us) server_negotiation.cc:407] Connection header received
0924 16:11:48.551221 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:48.551224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:48.551278 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:48.551357 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:48.551661 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.552239 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.553084 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.553428 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.553519 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:48.553817 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:48.553894 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:48.553973 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:48.554061 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 16:11:48.554108 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 16:11:48.554296 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:48.550772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:33877 (local address 127.1.253.67:56151)
0924 16:11:48.551000 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:48.551014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:48.551133 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:48.551417 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:48.551423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:48.551434 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:11:48.551541 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:48.551551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.552349 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.552352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:48.552967 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:48.552975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:48.553544 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:48.553547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:48.553635 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:48.553667 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:11:48.553977 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:11:48.553987 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 16:11:48.554052 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 16:11:48.554135 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
W20260924 16:11:48.555053 4362 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 16:11:50.115078 3813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:50.115247 3813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:50.117853 3813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66: Sending DeleteTablet for 3 replicas of tablet 97f156ade85a4735902c59b014140e76
I20260924 16:11:50.118451 4095 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f156ade85a4735902c59b014140e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:50 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:57610
I20260924 16:11:50.118464 3952 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f156ade85a4735902c59b014140e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:50 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:42364
I20260924 16:11:50.118618 4239 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f156ade85a4735902c59b014140e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:50 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:56318
I20260924 16:11:50.118775 4428 tablet_replica.cc:330] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7: stopping tablet replica
I20260924 16:11:50.118897 4428 raft_consensus.cc:2229] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:11:50.118925 4430 tablet_replica.cc:330] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49: stopping tablet replica
I20260924 16:11:50.119042 4430 raft_consensus.cc:2229] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:50.119158 4430 raft_consensus.cc:2258] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:50.119225 4428 raft_consensus.cc:2258] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:50.119465 4429 tablet_replica.cc:330] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6: stopping tablet replica
I20260924 16:11:50.119602 4429 raft_consensus.cc:2229] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:50.119720 4429 raft_consensus.cc:2258] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:50.119786 4430 ts_tablet_manager.cc:1905] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:50.120412 4428 ts_tablet_manager.cc:1905] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:50.121857 4429 ts_tablet_manager.cc:1905] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:50.122581 4430 ts_tablet_manager.cc:1918] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:11:50.122723 4430 log.cc:1196] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/97f156ade85a4735902c59b014140e76
I20260924 16:11:50.123080 4430 ts_tablet_manager.cc:1939] T 97f156ade85a4735902c59b014140e76 P be6d362797114bff9b6107b151235f49: Deleting consensus metadata
I20260924 16:11:50.123499 4428 ts_tablet_manager.cc:1918] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:11:50.123690 4428 log.cc:1196] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/97f156ade85a4735902c59b014140e76
I20260924 16:11:50.123905 3799 catalog_manager.cc:4841] TS be6d362797114bff9b6107b151235f49 (127.1.253.67:45313): tablet 97f156ade85a4735902c59b014140e76 (table test-table [id=26c8e6559f61479181b48d3a0827d893]) successfully deleted
I20260924 16:11:50.124068 4428 ts_tablet_manager.cc:1939] T 97f156ade85a4735902c59b014140e76 P c9010c9ecf154abe86b2ab1834cde9f7: Deleting consensus metadata
I20260924 16:11:50.124631 4429 ts_tablet_manager.cc:1918] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:11:50.124732 4429 log.cc:1196] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/wal/wals/97f156ade85a4735902c59b014140e76
I20260924 16:11:50.124856 3798 catalog_manager.cc:4841] TS c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877): tablet 97f156ade85a4735902c59b014140e76 (table test-table [id=26c8e6559f61479181b48d3a0827d893]) successfully deleted
I20260924 16:11:50.125056 4429 ts_tablet_manager.cc:1939] T 97f156ade85a4735902c59b014140e76 P 0603fc7358f84df0bef56cf51fb760b6: Deleting consensus metadata
I20260924 16:11:50.125684 3797 catalog_manager.cc:4841] TS 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767): tablet 97f156ade85a4735902c59b014140e76 (table test-table [id=26c8e6559f61479181b48d3a0827d893]) successfully deleted
Sep 24 16:11:50 dist-test-slave-6wvm krb5kdc[3768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:11:50 dist-test-slave-6wvm krb5kdc[3768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:50.155791 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.141528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.1.253.126:42659)
0924 16:11:50.141804 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:50.141812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:50.141890 (+ 78us) server_negotiation.cc:407] Connection header received
0924 16:11:50.142017 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:50.142019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:50.142065 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:50.142146 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:50.143017 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.143603 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.144707 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.144944 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.145029 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:50.152752 (+ 7723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:50.152756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:50.152758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:50.152785 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:50.154125 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:50.154600 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:50.154603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:50.154605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:50.154694 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:50.154942 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:50.154945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:50.154946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:50.155127 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:50.155251 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:50.155470 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 16:11:50.155572 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260924 16:11:50.158416 3813 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:11:50.158803 3813 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:11:50.163661 3952 tablet_service.cc:1467] Processing CreateTablet for tablet dc59132addcd48e7b00a4b33b8d563f9 (DEFAULT_TABLE table=test-table [id=5393c768e0ee4b54b1e68797c1c46daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:50.163913 4095 tablet_service.cc:1467] Processing CreateTablet for tablet dc59132addcd48e7b00a4b33b8d563f9 (DEFAULT_TABLE table=test-table [id=5393c768e0ee4b54b1e68797c1c46daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:50.163931 4239 tablet_service.cc:1467] Processing CreateTablet for tablet dc59132addcd48e7b00a4b33b8d563f9 (DEFAULT_TABLE table=test-table [id=5393c768e0ee4b54b1e68797c1c46daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:50.164018 3952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc59132addcd48e7b00a4b33b8d563f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:50.164186 4239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc59132addcd48e7b00a4b33b8d563f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:50.164216 4095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc59132addcd48e7b00a4b33b8d563f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:50.166527 4439 tablet_bootstrap.cc:492] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7: Bootstrap starting.
I20260924 16:11:50.166927 4440 tablet_bootstrap.cc:492] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6: Bootstrap starting.
I20260924 16:11:50.166930 4441 tablet_bootstrap.cc:492] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49: Bootstrap starting.
I20260924 16:11:50.167429 4439 tablet_bootstrap.cc:654] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:50.167802 4440 tablet_bootstrap.cc:654] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:50.167802 4441 tablet_bootstrap.cc:654] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:50.168656 4439 tablet_bootstrap.cc:492] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7: No bootstrap required, opened a new log
I20260924 16:11:50.168762 4439 ts_tablet_manager.cc:1397] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:11:50.168941 4440 tablet_bootstrap.cc:492] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6: No bootstrap required, opened a new log
I20260924 16:11:50.169035 4440 ts_tablet_manager.cc:1397] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:11:50.169241 4439 raft_consensus.cc:348] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.169378 4439 raft_consensus.cc:374] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:50.169423 4439 raft_consensus.cc:729] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9010c9ecf154abe86b2ab1834cde9f7, State: Initialized, Role: FOLLOWER
I20260924 16:11:50.169462 4440 raft_consensus.cc:348] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.169592 4440 raft_consensus.cc:374] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:50.169632 4440 raft_consensus.cc:729] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0603fc7358f84df0bef56cf51fb760b6, State: Initialized, Role: FOLLOWER
I20260924 16:11:50.169571 4439 consensus_queue.cc:260] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.169770 4440 consensus_queue.cc:260] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.169955 4439 ts_tablet_manager.cc:1428] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 16:11:50.170100 4440 ts_tablet_manager.cc:1428] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:11:50.170634 4441 tablet_bootstrap.cc:492] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49: No bootstrap required, opened a new log
I20260924 16:11:50.170728 4441 ts_tablet_manager.cc:1397] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 16:11:50.171216 4441 raft_consensus.cc:348] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.171321 4441 raft_consensus.cc:374] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:50.171370 4441 raft_consensus.cc:729] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6d362797114bff9b6107b151235f49, State: Initialized, Role: FOLLOWER
I20260924 16:11:50.171504 4441 consensus_queue.cc:260] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.171768 4441 ts_tablet_manager.cc:1428] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 16:11:50.434630 4423 raft_consensus.cc:482] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:50.434793 4423 raft_consensus.cc:504] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.435278 4423 leader_election.cc:288] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767), c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877)
I20260924 16:11:50.435853 4115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc59132addcd48e7b00a4b33b8d563f9" candidate_uuid: "be6d362797114bff9b6107b151235f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" is_pre_election: true
I20260924 16:11:50.435850 3972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc59132addcd48e7b00a4b33b8d563f9" candidate_uuid: "be6d362797114bff9b6107b151235f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0603fc7358f84df0bef56cf51fb760b6" is_pre_election: true
I20260924 16:11:50.436026 3972 raft_consensus.cc:2454] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be6d362797114bff9b6107b151235f49 in term 0.
I20260924 16:11:50.436026 4115 raft_consensus.cc:2454] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be6d362797114bff9b6107b151235f49 in term 0.
I20260924 16:11:50.436383 4190 leader_election.cc:302] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0603fc7358f84df0bef56cf51fb760b6, be6d362797114bff9b6107b151235f49, c9010c9ecf154abe86b2ab1834cde9f7; no voters:
I20260924 16:11:50.436566 4423 raft_consensus.cc:2781] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:50.436627 4423 raft_consensus.cc:482] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:50.436674 4423 raft_consensus.cc:3037] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:50.437760 4423 raft_consensus.cc:504] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.438133 4423 leader_election.cc:288] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [CANDIDATE]: Term 1 election: Requested vote from peers 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767), c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877)
I20260924 16:11:50.438524 3972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc59132addcd48e7b00a4b33b8d563f9" candidate_uuid: "be6d362797114bff9b6107b151235f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0603fc7358f84df0bef56cf51fb760b6"
I20260924 16:11:50.438591 4115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc59132addcd48e7b00a4b33b8d563f9" candidate_uuid: "be6d362797114bff9b6107b151235f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9010c9ecf154abe86b2ab1834cde9f7"
I20260924 16:11:50.438643 3972 raft_consensus.cc:3037] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:50.438751 4115 raft_consensus.cc:3037] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:50.439783 3972 raft_consensus.cc:2454] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be6d362797114bff9b6107b151235f49 in term 1.
I20260924 16:11:50.439926 4115 raft_consensus.cc:2454] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be6d362797114bff9b6107b151235f49 in term 1.
I20260924 16:11:50.440137 4190 leader_election.cc:302] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0603fc7358f84df0bef56cf51fb760b6, be6d362797114bff9b6107b151235f49; no voters:
I20260924 16:11:50.440410 4423 raft_consensus.cc:2781] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:50.440536 4423 raft_consensus.cc:686] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 1 LEADER]: Becoming Leader. State: Replica: be6d362797114bff9b6107b151235f49, State: Running, Role: LEADER
I20260924 16:11:50.440697 4423 consensus_queue.cc:237] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.442605 3813 catalog_manager.cc:5495] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 reported cstate change: term changed from 0 to 1, leader changed from <none> to be6d362797114bff9b6107b151235f49 (127.1.253.67). New cstate: current_term: 1 leader_uuid: "be6d362797114bff9b6107b151235f49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } health_report { overall_health: HEALTHY } } }
I20260924 16:11:50.472196 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.469273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.1.253.67:45313)
0924 16:11:50.469519 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:50.469526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:50.469538 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:11:50.469578 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:50.469580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:50.469624 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:50.469695 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:50.469916 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.470503 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.471337 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.471555 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.471631 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:50.471833 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:50.471901 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:50.471952 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:50.472033 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 16:11:50.472076 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":110,"threads_started":1}
I20260924 16:11:50.480908 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.478060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:34157 (local address 127.1.253.67:45313)
0924 16:11:50.478180 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:50.478183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:50.478267 (+ 84us) server_negotiation.cc:407] Connection header received
0924 16:11:50.478434 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:50.478436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:50.478479 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:50.478543 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:50.478746 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.479215 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.480008 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.480221 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.480294 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:50.480523 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:50.480582 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:50.480634 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:50.480712 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 16:11:50.480796 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":50}
I20260924 16:11:50.480930 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.477933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:45313 (local address 127.1.253.67:34157)
0924 16:11:50.478169 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:50.478181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:50.478319 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:50.478566 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:50.478568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:50.478575 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:11:50.478654 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:50.478658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.479300 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.479303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:50.479902 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:50.479910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.480312 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.480314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:50.480398 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:50.480423 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:11:50.480650 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:11:50.480654 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:11:50.480699 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 16:11:50.480797 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":18,"thread_start_us":91,"threads_started":1}
I20260924 16:11:50.486771 3972 raft_consensus.cc:1261] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Refusing update from remote peer be6d362797114bff9b6107b151235f49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:50.486771 4115 raft_consensus.cc:1261] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Refusing update from remote peer be6d362797114bff9b6107b151235f49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:50.487294 4423 consensus_queue.cc:1035] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:50.487453 4409 consensus_queue.cc:1035] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:50.522704 3811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:50.522879 3811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:50.525139 3811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66: Sending DeleteTablet for 3 replicas of tablet dc59132addcd48e7b00a4b33b8d563f9
I20260924 16:11:50.525800 3952 tablet_service.cc:1514] Processing DeleteTablet for tablet dc59132addcd48e7b00a4b33b8d563f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:50 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:42364
I20260924 16:11:50.525864 4095 tablet_service.cc:1514] Processing DeleteTablet for tablet dc59132addcd48e7b00a4b33b8d563f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:50 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:57610
I20260924 16:11:50.526022 4429 tablet_replica.cc:330] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6: stopping tablet replica
I20260924 16:11:50.526051 4428 tablet_replica.cc:330] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7: stopping tablet replica
I20260924 16:11:50.526163 4429 raft_consensus.cc:2229] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:50.526175 4428 raft_consensus.cc:2229] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:50.526319 4428 raft_consensus.cc:2258] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:50.526338 4429 raft_consensus.cc:2258] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:50.527132 4428 ts_tablet_manager.cc:1905] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:50.527572 4429 ts_tablet_manager.cc:1905] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:50.527527 3811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:11:50.527943 3811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:11:50.529675 4239 tablet_service.cc:1514] Processing DeleteTablet for tablet dc59132addcd48e7b00a4b33b8d563f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:50 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:56318
I20260924 16:11:50.529873 4430 tablet_replica.cc:330] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49: stopping tablet replica
I20260924 16:11:50.530016 4430 raft_consensus.cc:2229] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:11:50.530426 4430 raft_consensus.cc:2258] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:50.531068 4428 ts_tablet_manager.cc:1918] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:11:50.531190 4428 log.cc:1196] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/dc59132addcd48e7b00a4b33b8d563f9
I20260924 16:11:50.531250 4430 ts_tablet_manager.cc:1905] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:50.533358 4239 tablet_service.cc:1467] Processing CreateTablet for tablet a19c2e4fed3740a08158ce9217d1eb6a (DEFAULT_TABLE table=test-table [id=f0251dfae41b4999b84c7762ad22a7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:50.533608 4239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19c2e4fed3740a08158ce9217d1eb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:50.533636 4095 tablet_service.cc:1467] Processing CreateTablet for tablet a19c2e4fed3740a08158ce9217d1eb6a (DEFAULT_TABLE table=test-table [id=f0251dfae41b4999b84c7762ad22a7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:50.533809 4430 ts_tablet_manager.cc:1918] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:11:50.533876 4430 log.cc:1196] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/dc59132addcd48e7b00a4b33b8d563f9
I20260924 16:11:50.533905 4095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19c2e4fed3740a08158ce9217d1eb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:50.534124 4430 ts_tablet_manager.cc:1939] T dc59132addcd48e7b00a4b33b8d563f9 P be6d362797114bff9b6107b151235f49: Deleting consensus metadata
I20260924 16:11:50.534684 3952 tablet_service.cc:1467] Processing CreateTablet for tablet a19c2e4fed3740a08158ce9217d1eb6a (DEFAULT_TABLE table=test-table [id=f0251dfae41b4999b84c7762ad22a7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:11:50.534956 3952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19c2e4fed3740a08158ce9217d1eb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:50.536253 4439 tablet_bootstrap.cc:492] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7: Bootstrap starting.
I20260924 16:11:50.536628 4428 ts_tablet_manager.cc:1939] T dc59132addcd48e7b00a4b33b8d563f9 P c9010c9ecf154abe86b2ab1834cde9f7: Deleting consensus metadata
I20260924 16:11:50.537217 4439 tablet_bootstrap.cc:654] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:50.537340 4440 tablet_bootstrap.cc:492] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6: Bootstrap starting.
I20260924 16:11:50.537344 3798 catalog_manager.cc:4841] TS c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877): tablet dc59132addcd48e7b00a4b33b8d563f9 (table test-table [id=5393c768e0ee4b54b1e68797c1c46daf]) successfully deleted
I20260924 16:11:50.537384 4441 tablet_bootstrap.cc:492] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49: Bootstrap starting.
I20260924 16:11:50.538178 4440 tablet_bootstrap.cc:654] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:50.538321 4441 tablet_bootstrap.cc:654] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:50.538463 4439 tablet_bootstrap.cc:492] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7: No bootstrap required, opened a new log
I20260924 16:11:50.538535 4439 ts_tablet_manager.cc:1397] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:11:50.538987 4439 raft_consensus.cc:348] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.539111 4439 raft_consensus.cc:374] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:50.539146 4439 raft_consensus.cc:729] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9010c9ecf154abe86b2ab1834cde9f7, State: Initialized, Role: FOLLOWER
I20260924 16:11:50.539273 4439 consensus_queue.cc:260] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.539691 4439 ts_tablet_manager.cc:1428] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:11:50.540414 4441 tablet_bootstrap.cc:492] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49: No bootstrap required, opened a new log
I20260924 16:11:50.540504 4441 ts_tablet_manager.cc:1397] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 16:11:50.540918 4441 raft_consensus.cc:348] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.541033 4441 raft_consensus.cc:374] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:50.541065 4441 raft_consensus.cc:729] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6d362797114bff9b6107b151235f49, State: Initialized, Role: FOLLOWER
I20260924 16:11:50.541191 4441 consensus_queue.cc:260] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.541487 4441 ts_tablet_manager.cc:1428] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 16:11:50.539508 4429 ts_tablet_manager.cc:1918] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:11:50.541893 4429 log.cc:1196] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-0/wal/wals/dc59132addcd48e7b00a4b33b8d563f9
I20260924 16:11:50.542151 4429 ts_tablet_manager.cc:1939] T dc59132addcd48e7b00a4b33b8d563f9 P 0603fc7358f84df0bef56cf51fb760b6: Deleting consensus metadata
I20260924 16:11:50.542819 3797 catalog_manager.cc:4841] TS 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767): tablet dc59132addcd48e7b00a4b33b8d563f9 (table test-table [id=5393c768e0ee4b54b1e68797c1c46daf]) successfully deleted
I20260924 16:11:50.543246 3799 catalog_manager.cc:4841] TS be6d362797114bff9b6107b151235f49 (127.1.253.67:45313): tablet dc59132addcd48e7b00a4b33b8d563f9 (table test-table [id=5393c768e0ee4b54b1e68797c1c46daf]) successfully deleted
I20260924 16:11:50.544098 4440 tablet_bootstrap.cc:492] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6: No bootstrap required, opened a new log
I20260924 16:11:50.544214 4440 ts_tablet_manager.cc:1397] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260924 16:11:50.544541 4440 raft_consensus.cc:348] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.544655 4440 raft_consensus.cc:374] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:50.544695 4440 raft_consensus.cc:729] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0603fc7358f84df0bef56cf51fb760b6, State: Initialized, Role: FOLLOWER
I20260924 16:11:50.544826 4440 consensus_queue.cc:260] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.545104 4440 ts_tablet_manager.cc:1428] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 16:11:50.671650 4401 raft_consensus.cc:482] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:50.671815 4401 raft_consensus.cc:504] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.672251 4401 leader_election.cc:288] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767), be6d362797114bff9b6107b151235f49 (127.1.253.67:45313)
I20260924 16:11:50.672744 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c2e4fed3740a08158ce9217d1eb6a" candidate_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be6d362797114bff9b6107b151235f49" is_pre_election: true
I20260924 16:11:50.672734 3972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c2e4fed3740a08158ce9217d1eb6a" candidate_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0603fc7358f84df0bef56cf51fb760b6" is_pre_election: true
I20260924 16:11:50.672889 4259 raft_consensus.cc:2454] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9010c9ecf154abe86b2ab1834cde9f7 in term 0.
I20260924 16:11:50.672899 3972 raft_consensus.cc:2454] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9010c9ecf154abe86b2ab1834cde9f7 in term 0.
I20260924 16:11:50.672941 4423 raft_consensus.cc:482] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:11:50.673009 4423 raft_consensus.cc:504] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.673234 4046 leader_election.cc:302] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0603fc7358f84df0bef56cf51fb760b6, c9010c9ecf154abe86b2ab1834cde9f7; no voters:
I20260924 16:11:50.673395 4423 leader_election.cc:288] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767), c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877)
I20260924 16:11:50.673473 4401 raft_consensus.cc:2781] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:50.673523 4401 raft_consensus.cc:482] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:50.673568 4401 raft_consensus.cc:3037] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:50.673771 3972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c2e4fed3740a08158ce9217d1eb6a" candidate_uuid: "be6d362797114bff9b6107b151235f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0603fc7358f84df0bef56cf51fb760b6" is_pre_election: true
I20260924 16:11:50.673887 3972 raft_consensus.cc:2454] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be6d362797114bff9b6107b151235f49 in term 0.
I20260924 16:11:50.673866 4115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c2e4fed3740a08158ce9217d1eb6a" candidate_uuid: "be6d362797114bff9b6107b151235f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" is_pre_election: true
I20260924 16:11:50.674201 4190 leader_election.cc:302] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0603fc7358f84df0bef56cf51fb760b6, be6d362797114bff9b6107b151235f49; no voters:
I20260924 16:11:50.674480 4423 raft_consensus.cc:2781] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:11:50.674562 4423 raft_consensus.cc:482] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:11:50.674609 4423 raft_consensus.cc:3037] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:50.675182 4401 raft_consensus.cc:504] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.675446 4115 raft_consensus.cc:2379] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be6d362797114bff9b6107b151235f49 in current term 1: Already voted for candidate c9010c9ecf154abe86b2ab1834cde9f7 in this term.
I20260924 16:11:50.675555 4401 leader_election.cc:288] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [CANDIDATE]: Term 1 election: Requested vote from peers 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767), be6d362797114bff9b6107b151235f49 (127.1.253.67:45313)
I20260924 16:11:50.675900 4423 raft_consensus.cc:504] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.676191 3972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c2e4fed3740a08158ce9217d1eb6a" candidate_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0603fc7358f84df0bef56cf51fb760b6"
I20260924 16:11:50.676247 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c2e4fed3740a08158ce9217d1eb6a" candidate_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be6d362797114bff9b6107b151235f49"
I20260924 16:11:50.676278 4423 leader_election.cc:288] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [CANDIDATE]: Term 1 election: Requested vote from peers 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767), c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877)
I20260924 16:11:50.676317 3972 raft_consensus.cc:3037] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:50.676393 4259 raft_consensus.cc:2379] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c9010c9ecf154abe86b2ab1834cde9f7 in current term 1: Already voted for candidate be6d362797114bff9b6107b151235f49 in this term.
I20260924 16:11:50.676693 3971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c2e4fed3740a08158ce9217d1eb6a" candidate_uuid: "be6d362797114bff9b6107b151235f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0603fc7358f84df0bef56cf51fb760b6"
I20260924 16:11:50.676865 4115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19c2e4fed3740a08158ce9217d1eb6a" candidate_uuid: "be6d362797114bff9b6107b151235f49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9010c9ecf154abe86b2ab1834cde9f7"
I20260924 16:11:50.677021 4115 raft_consensus.cc:2379] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate be6d362797114bff9b6107b151235f49 in current term 1: Already voted for candidate c9010c9ecf154abe86b2ab1834cde9f7 in this term.
I20260924 16:11:50.677258 4191 leader_election.cc:302] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be6d362797114bff9b6107b151235f49; no voters: 0603fc7358f84df0bef56cf51fb760b6, c9010c9ecf154abe86b2ab1834cde9f7
I20260924 16:11:50.677452 4423 raft_consensus.cc:2726] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 16:11:50.677590 3972 raft_consensus.cc:2454] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9010c9ecf154abe86b2ab1834cde9f7 in term 1.
I20260924 16:11:50.677868 4046 leader_election.cc:302] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0603fc7358f84df0bef56cf51fb760b6, c9010c9ecf154abe86b2ab1834cde9f7; no voters: be6d362797114bff9b6107b151235f49
I20260924 16:11:50.678073 4401 raft_consensus.cc:2781] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:50.678300 4401 raft_consensus.cc:686] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 LEADER]: Becoming Leader. State: Replica: c9010c9ecf154abe86b2ab1834cde9f7, State: Running, Role: LEADER
I20260924 16:11:50.678499 4401 consensus_queue.cc:237] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } }
I20260924 16:11:50.680140 3811 catalog_manager.cc:5495] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66). New cstate: current_term: 1 leader_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9010c9ecf154abe86b2ab1834cde9f7" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 } health_report { overall_health: UNKNOWN } } }
I20260924 16:11:50.705061 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:50.701499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.1.253.66:33877)
0924 16:11:50.701787 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:50.701805 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:50.701834 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:11:50.701894 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:50.701897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:50.701950 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:50.702060 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:11:50.702405 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.703039 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.703983 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:50.704208 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:50.704322 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:50.704587 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:11:50.704668 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:11:50.704752 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:50.704845 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 16:11:50.704898 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":101,"threads_started":1}
I20260924 16:11:50.711381 3972 raft_consensus.cc:1261] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6 [term 1 FOLLOWER]: Refusing update from remote peer c9010c9ecf154abe86b2ab1834cde9f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:50.711796 4458 consensus_queue.cc:1035] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0603fc7358f84df0bef56cf51fb760b6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:11:50.711890 4259 raft_consensus.cc:1261] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Refusing update from remote peer c9010c9ecf154abe86b2ab1834cde9f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:11:50.712258 4458 consensus_queue.cc:1035] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be6d362797114bff9b6107b151235f49" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:11:50.725636 4410 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 16:11:52.277272 3811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:52.277420 3811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:11:52.279749 3811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e6cb9a2beac456fb289b247b4edfb66: Sending DeleteTablet for 3 replicas of tablet a19c2e4fed3740a08158ce9217d1eb6a
I20260924 16:11:52.280316 3952 tablet_service.cc:1514] Processing DeleteTablet for tablet a19c2e4fed3740a08158ce9217d1eb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:52 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:42364
I20260924 16:11:52.280438 4239 tablet_service.cc:1514] Processing DeleteTablet for tablet a19c2e4fed3740a08158ce9217d1eb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:52 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:56318
I20260924 16:11:52.280678 4486 tablet_replica.cc:330] T a19c2e4fed3740a08158ce9217d1eb6a P 0603fc7358f84df0bef56cf51fb760b6: stopping tablet replica
I20260924 16:11:52.280673 4095 tablet_service.cc:1514] Processing DeleteTablet for tablet a19c2e4fed3740a08158ce9217d1eb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:11:52 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:57610
I20260924 16:11:52.280869 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 3887
I20260924 16:11:52.281939 4488 tablet_replica.cc:330] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7: stopping tablet replica
I20260924 16:11:52.282112 4488 raft_consensus.cc:2229] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:11:52.282387 4488 raft_consensus.cc:2258] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:52.282666 4487 tablet_replica.cc:330] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49: stopping tablet replica
I20260924 16:11:52.282833 4487 raft_consensus.cc:2229] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:11:52.282961 4487 raft_consensus.cc:2258] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:11:52.283339 4488 ts_tablet_manager.cc:1905] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:52.283696 4487 ts_tablet_manager.cc:1905] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:11:52.285591 4488 ts_tablet_manager.cc:1918] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:11:52.285688 4488 log.cc:1196] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/a19c2e4fed3740a08158ce9217d1eb6a
I20260924 16:11:52.285964 4488 ts_tablet_manager.cc:1939] T a19c2e4fed3740a08158ce9217d1eb6a P c9010c9ecf154abe86b2ab1834cde9f7: Deleting consensus metadata
I20260924 16:11:52.286728 3798 catalog_manager.cc:4841] TS c9010c9ecf154abe86b2ab1834cde9f7 (127.1.253.66:33877): tablet a19c2e4fed3740a08158ce9217d1eb6a (table test-table [id=f0251dfae41b4999b84c7762ad22a7ee]) successfully deleted
I20260924 16:11:52.287216 4487 ts_tablet_manager.cc:1918] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:11:52.287302 4487 log.cc:1196] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/a19c2e4fed3740a08158ce9217d1eb6a
I20260924 16:11:52.287603 4487 ts_tablet_manager.cc:1939] T a19c2e4fed3740a08158ce9217d1eb6a P be6d362797114bff9b6107b151235f49: Deleting consensus metadata
I20260924 16:11:52.288379 3799 catalog_manager.cc:4841] TS be6d362797114bff9b6107b151235f49 (127.1.253.67:45313): tablet a19c2e4fed3740a08158ce9217d1eb6a (table test-table [id=f0251dfae41b4999b84c7762ad22a7ee]) successfully deleted
W20260924 16:11:52.291607 3797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:40767 (error 108)
I20260924 16:11:52.291818 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 4030
I20260924 16:11:52.293624 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:52.293105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40767 (local address 127.0.0.1:42392)
0924 16:11:52.293394 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:52.293492 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:40767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
W20260924 16:11:52.293779 3797 catalog_manager.cc:4584] TS 0603fc7358f84df0bef56cf51fb760b6 (127.1.253.65:40767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a19c2e4fed3740a08158ce9217d1eb6a: Network error: Client connection negotiation failed: client connection to 127.1.253.65:40767: connect: Connection refused (error 111)
I20260924 16:11:52.300859 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 4174
I20260924 16:11:52.308413 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 3781
2026-09-24T16:11:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7315 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:52 dist-test-slave-6wvm krb5kdc[4493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:52 dist-test-slave-6wvm krb5kdc[4493](info): set up 2 sockets
Sep 24 16:11:52 dist-test-slave-6wvm krb5kdc[4493](info): commencing operation
krb5kdc: starting...
W20260924 16:11:53.365938 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:11:53 dist-test-slave-6wvm krb5kdc[4493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:11:53.490712 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:38027
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:38027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:53.605760 4506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:53.606029 4506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:53.606071 4506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:53.610038 4506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:53.610096 4506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:53.610119 4506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:53.610139 4506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:53.610157 4506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:53.615453 4506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:38027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:38027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:53.616508 4506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:53.617570 4506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:53.626590 4514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:53.626555 4512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:53.626569 4511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:53.626727 4506 server_base.cc:1034] running on GCE node
I20260924 16:11:53.627420 4506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:53.628501 4506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:53.629712 4506 hybrid_clock.cc:648] HybridClock initialized: now 1790266313629693 us; error 33 us; skew 500 ppm
Sep 24 16:11:53 dist-test-slave-6wvm krb5kdc[4493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266313, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:53.632629 4506 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:11:53.633538 4506 webserver.cc:460] Webserver started at http://127.1.253.126:42467/ using document root <none> and password file <none>
I20260924 16:11:53.634102 4506 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:53.634194 4506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:53.634464 4506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:53.636482 4506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "273e306229c94d5b8349162b26378237"
format_stamp: "Formatted at 2026-09-24 16:11:53 on dist-test-slave-6wvm"
server_key: "bba80fcb76714779a1d72e56d9f64a52"
server_key_iv: "6117d8655119a014fd2aec6a738c4a86"
server_key_version: "encryptionkey@0"
I20260924 16:11:53.637053 4506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "273e306229c94d5b8349162b26378237"
format_stamp: "Formatted at 2026-09-24 16:11:53 on dist-test-slave-6wvm"
server_key: "bba80fcb76714779a1d72e56d9f64a52"
server_key_iv: "6117d8655119a014fd2aec6a738c4a86"
server_key_version: "encryptionkey@0"
I20260924 16:11:53.640756 4506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:11:53.643437 4521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:53.644760 4506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:11:53.644937 4506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "273e306229c94d5b8349162b26378237"
format_stamp: "Formatted at 2026-09-24 16:11:53 on dist-test-slave-6wvm"
server_key: "bba80fcb76714779a1d72e56d9f64a52"
server_key_iv: "6117d8655119a014fd2aec6a738c4a86"
server_key_version: "encryptionkey@0"
I20260924 16:11:53.645051 4506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:53.664635 4506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:53.668120 4506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:53.668326 4506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:53.675710 4506 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:38027
I20260924 16:11:53.675735 4573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:38027 every 8 connection(s)
I20260924 16:11:53.676702 4506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:53.677189 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 4506
I20260924 16:11:53.677315 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:11:53.677603 2037 external_mini_cluster.cc:1428] Setting key 918225e15c5b6d538bfd047cf3dc6078
I20260924 16:11:53.680016 4574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:53.686693 4574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237: Bootstrap starting.
Sep 24 16:11:53 dist-test-slave-6wvm krb5kdc[4493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:53.689563 4574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:53.690510 4574 log.cc:824] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:53.692598 4574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237: No bootstrap required, opened a new log
I20260924 16:11:53.692824 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:53.679035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33452 (local address 127.1.253.126:38027)
0924 16:11:53.679474 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:53.679493 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:53.679536 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:11:53.680397 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:53.680432 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:53.680843 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:53.681210 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:53.681884 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:53.683427 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:53.684320 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:53.685023 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:53.685181 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:53.687833 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:53.687841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:53.687861 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:53.687900 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:53.689833 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:53.690276 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:53.690285 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:53.690293 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:53.690451 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:53.690748 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:53.690751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:53.690753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:53.691219 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:53.691447 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:53.691755 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 16:11:53.692047 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":110,"threads_started":1}
I20260924 16:11:53.696347 4574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273e306229c94d5b8349162b26378237" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38027 } }
I20260924 16:11:53.696594 4574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:53.696686 4574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273e306229c94d5b8349162b26378237, State: Initialized, Role: FOLLOWER
I20260924 16:11:53.697113 4574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273e306229c94d5b8349162b26378237" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38027 } }
I20260924 16:11:53.697260 4574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:53.697345 4574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:53.697463 4574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:53.698432 4574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273e306229c94d5b8349162b26378237" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38027 } }
I20260924 16:11:53.698779 4574 leader_election.cc:302] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273e306229c94d5b8349162b26378237; no voters:
I20260924 16:11:53.699120 4574 leader_election.cc:288] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:53.699270 4579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:53.699548 4579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [term 1 LEADER]: Becoming Leader. State: Replica: 273e306229c94d5b8349162b26378237, State: Running, Role: LEADER
I20260924 16:11:53.699966 4579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273e306229c94d5b8349162b26378237" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38027 } }
I20260924 16:11:53.700484 4574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:53.701676 4580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273e306229c94d5b8349162b26378237. Latest consensus state: current_term: 1 leader_uuid: "273e306229c94d5b8349162b26378237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273e306229c94d5b8349162b26378237" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38027 } } }
I20260924 16:11:53.701807 4580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:53.701983 4581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "273e306229c94d5b8349162b26378237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273e306229c94d5b8349162b26378237" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38027 } } }
I20260924 16:11:53.702163 4581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237 [sys.catalog]: This master's current role is: LEADER
W20260924 16:11:53.705176 4594 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:11:53.705279 4594 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:11:53.705353 4589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:53.706079 4589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:53.711333 4589 catalog_manager.cc:1347] Generated new cluster ID: 7b6fbffbf6fe45408b6c6f734784765a
I20260924 16:11:53.711408 4589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:53.724181 4589 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:53.725160 4589 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:53.741317 4589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 273e306229c94d5b8349162b26378237: Generated new TSK 0
I20260924 16:11:53.742060 4589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:11:53.777287 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38027
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:53.891991 4600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:53.892289 4600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:53.892333 4600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:53.896448 4600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:53.896509 4600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:53.896610 4600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:11:53.901996 4600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:53.903167 4600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:53.904217 4600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:53.913980 4606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:53.913949 4608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:53.913982 4605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:53.914105 4600 server_base.cc:1034] running on GCE node
I20260924 16:11:53.914716 4600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:53.915372 4600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:53.916551 4600 hybrid_clock.cc:648] HybridClock initialized: now 1790266313916536 us; error 26 us; skew 500 ppm
Sep 24 16:11:53 dist-test-slave-6wvm krb5kdc[4493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266313, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:53.919497 4600 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:11:53.920483 4600 webserver.cc:460] Webserver started at http://127.1.253.65:39131/ using document root <none> and password file <none>
I20260924 16:11:53.921049 4600 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:53.921114 4600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:53.921368 4600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:53.923614 4600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "5aa49b42ef584d93a1a15752c387ae1b"
format_stamp: "Formatted at 2026-09-24 16:11:53 on dist-test-slave-6wvm"
server_key: "2954997f208e46a125bbeafb17b87915"
server_key_iv: "fef0edbcf4c6285ac468201aeb4fc5d9"
server_key_version: "encryptionkey@0"
I20260924 16:11:53.924188 4600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "5aa49b42ef584d93a1a15752c387ae1b"
format_stamp: "Formatted at 2026-09-24 16:11:53 on dist-test-slave-6wvm"
server_key: "2954997f208e46a125bbeafb17b87915"
server_key_iv: "fef0edbcf4c6285ac468201aeb4fc5d9"
server_key_version: "encryptionkey@0"
I20260924 16:11:53.928089 4600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:11:53.930678 4615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:53.931951 4600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:11:53.932098 4600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "5aa49b42ef584d93a1a15752c387ae1b"
format_stamp: "Formatted at 2026-09-24 16:11:53 on dist-test-slave-6wvm"
server_key: "2954997f208e46a125bbeafb17b87915"
server_key_iv: "fef0edbcf4c6285ac468201aeb4fc5d9"
server_key_version: "encryptionkey@0"
I20260924 16:11:53.932214 4600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:53.956208 4600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:53.959517 4600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:53.959738 4600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:53.960418 4600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:53.961325 4600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:53.961406 4600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:53.961484 4600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:53.961536 4600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:53.970713 4600 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:36391
I20260924 16:11:53.970732 4728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:36391 every 8 connection(s)
I20260924 16:11:53.971635 4600 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:11:53.974097 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 4600
I20260924 16:11:53.974187 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:11:53.974514 2037 external_mini_cluster.cc:1428] Setting key 037eb3550aa46c8b0f91c0d13d92533f
Sep 24 16:11:53 dist-test-slave-6wvm krb5kdc[4493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266313, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:11:53.986268 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:53.973637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:40757 (local address 127.1.253.126:38027)
0924 16:11:53.973809 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:53.973813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:53.974719 (+ 906us) server_negotiation.cc:407] Connection header received
0924 16:11:53.976333 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:53.976337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:53.976426 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:53.976524 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:53.977743 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:53.978353 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:53.979509 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:53.979760 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:53.979873 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:53.982455 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:53.982457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:53.982459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:53.982483 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:53.984062 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:53.984564 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:53.984568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:53.984570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:53.984665 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:53.985037 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:53.985047 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:53.985049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:53.985226 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:53.985348 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:53.985869 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 16:11:53.986070 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
I20260924 16:11:53.987159 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:53.974019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38027 (local address 127.1.253.65:40757)
0924 16:11:53.974551 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:53.974606 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:53.976124 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:53.976695 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:53.976702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:53.977064 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:53.977576 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:53.977590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:53.978479 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:53.978489 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:53.979374 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:53.979384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:53.979928 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:53.979932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:53.980110 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:53.980752 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:53.980769 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:53.982213 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:53.984177 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:53.984183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:53.984199 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:53.984455 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:53.984761 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:53.984765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:53.984767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:53.984907 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:53.985346 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:53.985353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:53.985630 (+ 277us) client_negotiation.cc:769] Sending connection context
0924 16:11:53.985828 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 16:11:53.986172 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":179,"threads_started":1}
I20260924 16:11:53.988441 4729 heartbeater.cc:347] Connected to a master server at 127.1.253.126:38027
I20260924 16:11:53.988708 4729 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:53.989212 4729 heartbeater.cc:511] Master 127.1.253.126:38027 requested a full tablet report, sending...
I20260924 16:11:53.991011 4538 ts_manager.cc:194] Registered new tserver with Master: 5aa49b42ef584d93a1a15752c387ae1b (127.1.253.65:36391)
I20260924 16:11:53.992964 4538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:40757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:11:54.003525 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38027
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:54.119627 4734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:54.119889 4734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:54.119931 4734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:54.123921 4734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:54.123976 4734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:54.124034 4734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:11:54.129163 4734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:54.130302 4734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:54.131306 4734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:11:54.141350 4734 server_base.cc:1034] running on GCE node
W20260924 16:11:54.141321 4739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:54.141364 4742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:54.141309 4740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:54.142047 4734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:54.142812 4734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:54.143998 4734 hybrid_clock.cc:648] HybridClock initialized: now 1790266314143984 us; error 26 us; skew 500 ppm
Sep 24 16:11:54 dist-test-slave-6wvm krb5kdc[4493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266314, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:54.146903 4734 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:11:54.147800 4734 webserver.cc:460] Webserver started at http://127.1.253.66:38493/ using document root <none> and password file <none>
I20260924 16:11:54.148469 4734 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:54.148548 4734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:54.148739 4734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:54.151108 4734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "7d34fcdae0d44d038fc5ef582be43d89"
format_stamp: "Formatted at 2026-09-24 16:11:54 on dist-test-slave-6wvm"
server_key: "90d569c53de2988640b2d5739acc456b"
server_key_iv: "26270ccf4426ab51a993ef26f263a8d9"
server_key_version: "encryptionkey@0"
I20260924 16:11:54.151767 4734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "7d34fcdae0d44d038fc5ef582be43d89"
format_stamp: "Formatted at 2026-09-24 16:11:54 on dist-test-slave-6wvm"
server_key: "90d569c53de2988640b2d5739acc456b"
server_key_iv: "26270ccf4426ab51a993ef26f263a8d9"
server_key_version: "encryptionkey@0"
I20260924 16:11:54.155607 4734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:11:54.157943 4749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:54.159276 4734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:11:54.159430 4734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "7d34fcdae0d44d038fc5ef582be43d89"
format_stamp: "Formatted at 2026-09-24 16:11:54 on dist-test-slave-6wvm"
server_key: "90d569c53de2988640b2d5739acc456b"
server_key_iv: "26270ccf4426ab51a993ef26f263a8d9"
server_key_version: "encryptionkey@0"
I20260924 16:11:54.159529 4734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:54.181159 4734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:54.184744 4734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:54.184923 4734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:54.185595 4734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:54.186499 4734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:54.186547 4734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:54.186615 4734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:54.186655 4734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:54.196058 4734 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:34053
I20260924 16:11:54.196077 4863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:34053 every 8 connection(s)
I20260924 16:11:54.197016 4734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:11:54.200794 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 4734
I20260924 16:11:54.200922 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:11:54.201217 2037 external_mini_cluster.cc:1428] Setting key baff43ef17c8b2ac6a98ff59b0e66f41
Sep 24 16:11:54 dist-test-slave-6wvm krb5kdc[4493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266314, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:11:54.213495 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:54.199077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:44767 (local address 127.1.253.126:38027)
0924 16:11:54.199229 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:54.199233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:54.200142 (+ 909us) server_negotiation.cc:407] Connection header received
0924 16:11:54.201955 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:54.201960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:54.202018 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:54.202118 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:54.203697 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:54.204535 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:54.206390 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:54.206605 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:54.206687 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:54.209623 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:54.209630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:54.209637 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:54.209669 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:54.211368 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:54.211884 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:54.211887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:54.211889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:54.211981 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:54.212393 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:54.212397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:54.212398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:54.212640 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:54.212722 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:54.213203 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 16:11:54.213370 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:11:54.214535 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:54.199505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38027 (local address 127.1.253.66:44767)
0924 16:11:54.199968 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:54.200012 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:54.201700 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:54.202384 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:54.202434 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:54.203020 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:54.203536 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:54.203551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:54.204695 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:54.204702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:54.206260 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:54.206275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:54.206769 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:54.206777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:54.207021 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:54.207708 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:54.207733 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:54.209444 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:54.211483 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:54.211489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:54.211501 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:54.211770 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:54.212099 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:54.212102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:54.212104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:54.212256 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:54.212736 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:54.212741 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:54.212984 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 16:11:54.213147 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 16:11:54.213487 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
I20260924 16:11:54.215950 4864 heartbeater.cc:347] Connected to a master server at 127.1.253.126:38027
I20260924 16:11:54.216216 4864 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:54.216826 4864 heartbeater.cc:511] Master 127.1.253.126:38027 requested a full tablet report, sending...
I20260924 16:11:54.218055 4538 ts_manager.cc:194] Registered new tserver with Master: 7d34fcdae0d44d038fc5ef582be43d89 (127.1.253.66:34053)
I20260924 16:11:54.218730 4538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:44767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:11:54.232741 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38027
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:54.347574 4869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:54.347908 4869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:54.347985 4869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:54.352303 4869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:54.352401 4869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:54.352495 4869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:11:54.357981 4869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:54.359226 4869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:54.360370 4869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:54.370440 4874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:54.370436 4875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:54.370682 4869 server_base.cc:1034] running on GCE node
W20260924 16:11:54.370478 4877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:54.371148 4869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:54.371786 4869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:54.372985 4869 hybrid_clock.cc:648] HybridClock initialized: now 1790266314372961 us; error 53 us; skew 500 ppm
Sep 24 16:11:54 dist-test-slave-6wvm krb5kdc[4493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266314, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:54.375741 4869 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:11:54.376632 4869 webserver.cc:460] Webserver started at http://127.1.253.67:46121/ using document root <none> and password file <none>
I20260924 16:11:54.377202 4869 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:54.377293 4869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:54.377529 4869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:54.379621 4869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "719a3e0224144df58da5524f7146dd8d"
format_stamp: "Formatted at 2026-09-24 16:11:54 on dist-test-slave-6wvm"
server_key: "0a9dfd4dcab0bc8922b5db153e1a65e5"
server_key_iv: "3c2db612140d3b9a109d111e492ac2f3"
server_key_version: "encryptionkey@0"
I20260924 16:11:54.380199 4869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "719a3e0224144df58da5524f7146dd8d"
format_stamp: "Formatted at 2026-09-24 16:11:54 on dist-test-slave-6wvm"
server_key: "0a9dfd4dcab0bc8922b5db153e1a65e5"
server_key_iv: "3c2db612140d3b9a109d111e492ac2f3"
server_key_version: "encryptionkey@0"
I20260924 16:11:54.384075 4869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:11:54.386664 4884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:54.387923 4869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:11:54.388041 4869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "719a3e0224144df58da5524f7146dd8d"
format_stamp: "Formatted at 2026-09-24 16:11:54 on dist-test-slave-6wvm"
server_key: "0a9dfd4dcab0bc8922b5db153e1a65e5"
server_key_iv: "3c2db612140d3b9a109d111e492ac2f3"
server_key_version: "encryptionkey@0"
I20260924 16:11:54.388162 4869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:54.415226 4869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:54.418504 4869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:54.418721 4869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:54.419379 4869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:54.420274 4869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:54.420348 4869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:54.420418 4869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:54.420466 4869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:54.429689 4869 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:45599
I20260924 16:11:54.429741 4999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:45599 every 8 connection(s)
I20260924 16:11:54.430773 4869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:11:54.439929 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 4869
I20260924 16:11:54.440026 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:11:54.440340 2037 external_mini_cluster.cc:1428] Setting key 20b7d767e09a96a3089ff13f14304fcf
Sep 24 16:11:54 dist-test-slave-6wvm krb5kdc[4493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266314, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:54.444713 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:54.432668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:39139 (local address 127.1.253.126:38027)
0924 16:11:54.432802 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:54.432805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:54.433609 (+ 804us) server_negotiation.cc:407] Connection header received
0924 16:11:54.435063 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:54.435066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:54.435139 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:54.435216 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:54.436490 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:54.437065 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:54.438269 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:54.438616 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:54.438744 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:54.441209 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:54.441212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:54.441214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:54.441240 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:54.442841 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:54.443382 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:54.443385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:54.443386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:54.443459 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:54.443762 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:54.443764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:54.443765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:54.443928 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:54.444001 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:54.444508 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 16:11:54.444599 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:11:54.445669 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:54.433005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38027 (local address 127.1.253.67:39139)
0924 16:11:54.433453 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:54.433498 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:54.434842 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:54.435402 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:54.435409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:54.435790 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:54.436338 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:54.436354 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:54.437222 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:54.437225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:54.438095 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:54.438105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:54.438786 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:54.438791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:54.438959 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:54.439500 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:54.439515 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:54.441028 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:54.442966 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:54.442971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:54.442982 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:54.443269 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:54.443541 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:54.443544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:54.443545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:54.443680 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:54.444059 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:54.444065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:54.444299 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 16:11:54.444464 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 16:11:54.444710 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
I20260924 16:11:54.446739 5000 heartbeater.cc:347] Connected to a master server at 127.1.253.126:38027
I20260924 16:11:54.446936 5000 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:54.447353 5000 heartbeater.cc:511] Master 127.1.253.126:38027 requested a full tablet report, sending...
I20260924 16:11:54.448328 4538 ts_manager.cc:194] Registered new tserver with Master: 719a3e0224144df58da5524f7146dd8d (127.1.253.67:45599)
I20260924 16:11:54.448897 4538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:39139
I20260924 16:11:54.454331 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:11:54.466995 4577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:11:54.465525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.1.253.126:38027)
0924 16:11:54.465700 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:54.465706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:54.465855 (+ 149us) server_negotiation.cc:407] Connection header received
0924 16:11:54.465962 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:54.465964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:54.466005 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:54.466070 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:54.466862 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33456: BlockingRecv error: recv got EOF from 127.0.0.1:33456 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:11:54.467993 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 4600
I20260924 16:11:54.473960 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 4734
I20260924 16:11:54.480180 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 4869
I20260924 16:11:54.485950 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 4506
2026-09-24T16:11:54Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2176 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:54 dist-test-slave-6wvm krb5kdc[5012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:54 dist-test-slave-6wvm krb5kdc[5012](info): set up 2 sockets
Sep 24 16:11:54 dist-test-slave-6wvm krb5kdc[5012](info): commencing operation
krb5kdc: starting...
W20260924 16:11:55.530079 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:11:55 dist-test-slave-6wvm krb5kdc[5012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:11:55.652890 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:35459
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:34421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:35459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:55.767288 5026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:55.767575 5026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:55.767613 5026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:55.771840 5026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:55.771898 5026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:55.771917 5026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:55.771934 5026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:55.771948 5026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:55.777120 5026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:34421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:35459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:35459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:55.778152 5026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:55.779261 5026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:55.788323 5034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:55.788313 5031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:55.788561 5026 server_base.cc:1034] running on GCE node
W20260924 16:11:55.788313 5032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:55.789319 5026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:55.790588 5026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:55.791790 5026 hybrid_clock.cc:648] HybridClock initialized: now 1790266315791744 us; error 59 us; skew 500 ppm
Sep 24 16:11:55 dist-test-slave-6wvm krb5kdc[5012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:55.806571 5026 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:11:55.807698 5026 webserver.cc:460] Webserver started at http://127.1.253.126:44211/ using document root <none> and password file <none>
I20260924 16:11:55.808362 5026 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:55.808455 5026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:55.808699 5026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:55.810956 5026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "1689b5a3fedd4f80b224a9d58c1d815f"
format_stamp: "Formatted at 2026-09-24 16:11:55 on dist-test-slave-6wvm"
server_key: "9c2368a4ab13899868036dc85a8b8b37"
server_key_iv: "26e0f437fd0d53b1cfc8a2a2942a6b79"
server_key_version: "encryptionkey@0"
I20260924 16:11:55.811578 5026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "1689b5a3fedd4f80b224a9d58c1d815f"
format_stamp: "Formatted at 2026-09-24 16:11:55 on dist-test-slave-6wvm"
server_key: "9c2368a4ab13899868036dc85a8b8b37"
server_key_iv: "26e0f437fd0d53b1cfc8a2a2942a6b79"
server_key_version: "encryptionkey@0"
I20260924 16:11:55.815399 5026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 16:11:55.817792 5041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:55.819118 5026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:11:55.819260 5026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "1689b5a3fedd4f80b224a9d58c1d815f"
format_stamp: "Formatted at 2026-09-24 16:11:55 on dist-test-slave-6wvm"
server_key: "9c2368a4ab13899868036dc85a8b8b37"
server_key_iv: "26e0f437fd0d53b1cfc8a2a2942a6b79"
server_key_version: "encryptionkey@0"
I20260924 16:11:55.819370 5026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:55.836453 5026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:55.839516 5026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:55.839740 5026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:55.846894 5026 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:35459
I20260924 16:11:55.847208 5096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:35459 every 8 connection(s)
I20260924 16:11:55.847879 5026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:55.849598 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 5026
I20260924 16:11:55.849735 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:11:55.849989 2037 external_mini_cluster.cc:1428] Setting key b609428e8139a3b2422947e270a1a11d
I20260924 16:11:55.851052 5097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:55.858033 5097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f: Bootstrap starting.
Sep 24 16:11:55 dist-test-slave-6wvm krb5kdc[5012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:55.860750 5097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:55.861496 5097 log.cc:824] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:55.863974 5097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f: No bootstrap required, opened a new log
I20260924 16:11:55.864884 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.851367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.1.253.126:35459)
0924 16:11:55.851794 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:55.851810 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:55.851847 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:11:55.852492 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:55.852508 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:55.852828 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:55.853111 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:55.854077 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.855226 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.856386 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.856834 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.856960 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:55.859771 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:55.859776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:55.859790 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:55.859817 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:55.861625 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:55.862147 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:55.862152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:55.862156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:55.862333 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:55.862635 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:55.862638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:55.862640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:55.863162 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:55.863352 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:55.863605 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 16:11:55.863918 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":139,"threads_started":1}
I20260924 16:11:55.867825 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:11:55.867995 5097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1689b5a3fedd4f80b224a9d58c1d815f" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35459 } }
I20260924 16:11:55.868217 5097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:55.868291 5097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1689b5a3fedd4f80b224a9d58c1d815f, State: Initialized, Role: FOLLOWER
I20260924 16:11:55.868707 5097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1689b5a3fedd4f80b224a9d58c1d815f" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35459 } }
I20260924 16:11:55.868826 5097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:55.868911 5097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:55.869014 5097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:55.870090 5097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1689b5a3fedd4f80b224a9d58c1d815f" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35459 } }
I20260924 16:11:55.870515 5097 leader_election.cc:302] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1689b5a3fedd4f80b224a9d58c1d815f; no voters:
I20260924 16:11:55.870922 5097 leader_election.cc:288] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:55.871026 5104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:55.871313 5104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [term 1 LEADER]: Becoming Leader. State: Replica: 1689b5a3fedd4f80b224a9d58c1d815f, State: Running, Role: LEADER
I20260924 16:11:55.871613 5104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1689b5a3fedd4f80b224a9d58c1d815f" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35459 } }
I20260924 16:11:55.872239 5097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:55.873270 5105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1689b5a3fedd4f80b224a9d58c1d815f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1689b5a3fedd4f80b224a9d58c1d815f" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35459 } } }
I20260924 16:11:55.873404 5105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:55.873821 5106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1689b5a3fedd4f80b224a9d58c1d815f. Latest consensus state: current_term: 1 leader_uuid: "1689b5a3fedd4f80b224a9d58c1d815f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1689b5a3fedd4f80b224a9d58c1d815f" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35459 } } }
I20260924 16:11:55.873993 5106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f [sys.catalog]: This master's current role is: LEADER
W20260924 16:11:55.877619 5120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:11:55.877717 5120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:11:55.877847 5113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:55.878635 5113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:55.884260 5113 catalog_manager.cc:1347] Generated new cluster ID: 49369ef900da4356987bad514f22c4c1
I20260924 16:11:55.884368 5113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:55.892154 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:55.886341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.1.253.126:35459)
0924 16:11:55.886447 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:55.886452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:55.886814 (+ 362us) server_negotiation.cc:407] Connection header received
0924 16:11:55.886953 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:55.886956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:55.887005 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:55.887100 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:55.887729 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.888526 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.889716 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:55.890044 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:55.890139 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:55.891118 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:55.891121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:55.891124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:11:55.891674 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:55.891709 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:11:55.891728 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:55.891815 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:55.891935 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 16:11:55.891997 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 16:11:55.904469 5113 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:55.905756 5113 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:55.918648 5113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1689b5a3fedd4f80b224a9d58c1d815f: Generated new TSK 0
I20260924 16:11:55.919588 5113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:11:55.933789 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 5026
2026-09-24T16:11:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1446 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:55 dist-test-slave-6wvm krb5kdc[5131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:55 dist-test-slave-6wvm krb5kdc[5131](info): set up 2 sockets
Sep 24 16:11:55 dist-test-slave-6wvm krb5kdc[5131](info): commencing operation
krb5kdc: starting...
W20260924 16:11:56.990056 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:11:57 dist-test-slave-6wvm krb5kdc[5131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:11:57.121850 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:43035
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:57.235340 5144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:57.235651 5144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:57.235729 5144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:57.239851 5144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:57.239935 5144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:57.239974 5144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:57.240027 5144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:57.240065 5144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:57.245296 5144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:43035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:43035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:57.246476 5144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:57.247552 5144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:57.256387 5149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:57.256387 5150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:57.256438 5144 server_base.cc:1034] running on GCE node
W20260924 16:11:57.256409 5152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:57.257231 5144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:57.258520 5144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:57.259686 5144 hybrid_clock.cc:648] HybridClock initialized: now 1790266317259658 us; error 38 us; skew 500 ppm
Sep 24 16:11:57 dist-test-slave-6wvm krb5kdc[5131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266317, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:57.273965 5144 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:11:57.275048 5144 webserver.cc:460] Webserver started at http://127.1.253.126:37165/ using document root <none> and password file <none>
I20260924 16:11:57.275632 5144 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:57.275697 5144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:57.275878 5144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:57.277976 5144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "93b0b507e5d94336837ebfcc05f95246"
format_stamp: "Formatted at 2026-09-24 16:11:57 on dist-test-slave-6wvm"
server_key: "dc4d9258d1ec921f7e07fa0b6a91c1a2"
server_key_iv: "55f28fe457d5ba9a1f5322e03f107a86"
server_key_version: "encryptionkey@0"
I20260924 16:11:57.278589 5144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "93b0b507e5d94336837ebfcc05f95246"
format_stamp: "Formatted at 2026-09-24 16:11:57 on dist-test-slave-6wvm"
server_key: "dc4d9258d1ec921f7e07fa0b6a91c1a2"
server_key_iv: "55f28fe457d5ba9a1f5322e03f107a86"
server_key_version: "encryptionkey@0"
I20260924 16:11:57.282533 5144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:11:57.284971 5159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:57.286336 5144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:11:57.286458 5144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "93b0b507e5d94336837ebfcc05f95246"
format_stamp: "Formatted at 2026-09-24 16:11:57 on dist-test-slave-6wvm"
server_key: "dc4d9258d1ec921f7e07fa0b6a91c1a2"
server_key_iv: "55f28fe457d5ba9a1f5322e03f107a86"
server_key_version: "encryptionkey@0"
I20260924 16:11:57.286552 5144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:57.304764 5144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:57.311846 5144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:57.312052 5144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:57.321910 5144 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:43035
I20260924 16:11:57.322280 5213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:43035 every 8 connection(s)
I20260924 16:11:57.323454 5144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:57.327380 5214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:57.329288 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 5144
I20260924 16:11:57.329421 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:11:57.329797 2037 external_mini_cluster.cc:1428] Setting key f667b872fbc6b835542dd02140bbeb88
I20260924 16:11:57.336263 5214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246: Bootstrap starting.
I20260924 16:11:57.339637 5214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246: Neither blocks nor log segments found. Creating new log.
Sep 24 16:11:57 dist-test-slave-6wvm krb5kdc[5131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:57.340683 5214 log.cc:824] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:57.343679 5214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246: No bootstrap required, opened a new log
I20260924 16:11:57.346859 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:57.331284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.1.253.126:43035)
0924 16:11:57.331739 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.331760 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.331971 (+ 211us) server_negotiation.cc:407] Connection header received
0924 16:11:57.332852 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.332872 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.333211 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.333547 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.334393 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.335331 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.336234 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.336857 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.337041 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:57.340764 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:57.340770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:57.340793 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:57.340832 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:57.344011 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:57.344490 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:57.344495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:57.344499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:57.344607 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:57.344912 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:57.344915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:57.344916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:57.345290 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:57.345486 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:57.345782 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 16:11:57.346079 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":141,"threads_started":1}
I20260924 16:11:57.347790 5214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b0b507e5d94336837ebfcc05f95246" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43035 } }
I20260924 16:11:57.348090 5214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:57.348141 5214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b0b507e5d94336837ebfcc05f95246, State: Initialized, Role: FOLLOWER
I20260924 16:11:57.348623 5214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b0b507e5d94336837ebfcc05f95246" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43035 } }
I20260924 16:11:57.348802 5214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:57.348865 5214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:57.348974 5214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:57.349917 5214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b0b507e5d94336837ebfcc05f95246" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43035 } }
I20260924 16:11:57.350320 5214 leader_election.cc:302] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93b0b507e5d94336837ebfcc05f95246; no voters:
I20260924 16:11:57.350663 5214 leader_election.cc:288] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:57.350814 5219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:57.351058 5219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [term 1 LEADER]: Becoming Leader. State: Replica: 93b0b507e5d94336837ebfcc05f95246, State: Running, Role: LEADER
I20260924 16:11:57.351377 5219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b0b507e5d94336837ebfcc05f95246" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43035 } }
I20260924 16:11:57.351938 5214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:57.353070 5221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93b0b507e5d94336837ebfcc05f95246. Latest consensus state: current_term: 1 leader_uuid: "93b0b507e5d94336837ebfcc05f95246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b0b507e5d94336837ebfcc05f95246" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43035 } } }
I20260924 16:11:57.353206 5221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:57.353502 5220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93b0b507e5d94336837ebfcc05f95246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b0b507e5d94336837ebfcc05f95246" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43035 } } }
I20260924 16:11:57.353610 5220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246 [sys.catalog]: This master's current role is: LEADER
W20260924 16:11:57.357082 5234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:11:57.357187 5234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:11:57.357277 5228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:57.357946 5228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:57.364173 5228 catalog_manager.cc:1347] Generated new cluster ID: 2d9f13f354d84a929a5771ac2c862d87
I20260924 16:11:57.364244 5228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:57.374471 5228 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:57.375451 5228 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:57.381563 5228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93b0b507e5d94336837ebfcc05f95246: Generated new TSK 0
I20260924 16:11:57.382340 5228 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:11:57.416706 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43035
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:57.532349 5240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:57.532584 5240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:57.532634 5240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:57.536571 5240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:57.536626 5240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:57.536681 5240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:11:57.541725 5240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:57.542775 5240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:57.543768 5240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:57.553176 5245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:57.553216 5240 server_base.cc:1034] running on GCE node
W20260924 16:11:57.553153 5248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:57.553403 5246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:57.553910 5240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:57.554589 5240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:57.555775 5240 hybrid_clock.cc:648] HybridClock initialized: now 1790266317555761 us; error 24 us; skew 500 ppm
Sep 24 16:11:57 dist-test-slave-6wvm krb5kdc[5131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266317, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:57.559226 5240 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:11:57.560079 5240 webserver.cc:460] Webserver started at http://127.1.253.65:43539/ using document root <none> and password file <none>
I20260924 16:11:57.560672 5240 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:57.560765 5240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:57.561012 5240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:57.563174 5240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "767fd347190249d082350207d6aa44bb"
format_stamp: "Formatted at 2026-09-24 16:11:57 on dist-test-slave-6wvm"
server_key: "4fc8d0e33ed8bdf3b64ab6a6f0bab714"
server_key_iv: "9cfa47ef566fb588dc3ef2d7efc7ce4f"
server_key_version: "encryptionkey@0"
I20260924 16:11:57.563740 5240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "767fd347190249d082350207d6aa44bb"
format_stamp: "Formatted at 2026-09-24 16:11:57 on dist-test-slave-6wvm"
server_key: "4fc8d0e33ed8bdf3b64ab6a6f0bab714"
server_key_iv: "9cfa47ef566fb588dc3ef2d7efc7ce4f"
server_key_version: "encryptionkey@0"
I20260924 16:11:57.567456 5240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:11:57.569802 5255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:57.571123 5240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:11:57.571266 5240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "767fd347190249d082350207d6aa44bb"
format_stamp: "Formatted at 2026-09-24 16:11:57 on dist-test-slave-6wvm"
server_key: "4fc8d0e33ed8bdf3b64ab6a6f0bab714"
server_key_iv: "9cfa47ef566fb588dc3ef2d7efc7ce4f"
server_key_version: "encryptionkey@0"
I20260924 16:11:57.571372 5240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:57.589057 5240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:57.594796 5240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:57.595008 5240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:57.595690 5240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:57.596513 5240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:57.596560 5240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:57.596601 5240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:57.596628 5240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:57.605062 5240 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:42585
I20260924 16:11:57.605147 5372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:42585 every 8 connection(s)
I20260924 16:11:57.606017 5240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:11:57.613590 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 5240
I20260924 16:11:57.613772 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:11:57.614087 2037 external_mini_cluster.cc:1428] Setting key 65e2fac914f297d99c609c8cda909d3e
Sep 24 16:11:57 dist-test-slave-6wvm krb5kdc[5131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266317, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:57.623106 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:57.607999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:50291 (local address 127.1.253.126:43035)
0924 16:11:57.608157 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.608160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.609005 (+ 845us) server_negotiation.cc:407] Connection header received
0924 16:11:57.610468 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.610472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.610532 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.610645 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.611812 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.612359 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.613690 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.614005 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.614101 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:57.617760 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:57.617766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:57.617770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:57.617804 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:57.620790 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:57.621325 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:57.621332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:57.621337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:57.621435 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:57.621773 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:57.621778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:57.621781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:57.622058 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:57.622177 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:57.622758 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 16:11:57.622901 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:11:57.623853 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:57.608393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:43035 (local address 127.1.253.65:50291)
0924 16:11:57.608850 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:57.608893 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:57.610175 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:57.610826 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:57.610833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:57.611194 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:57.611666 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:57.611678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.612489 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.612492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:57.613527 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:57.613540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.614179 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.614185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:57.614417 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:57.615807 (+ 1390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:57.615824 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:57.617586 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:57.620928 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:57.620933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:57.620946 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:57.621216 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:57.621539 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:57.621542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:57.621543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:57.621674 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:57.622184 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:57.622189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:57.622509 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 16:11:57.622675 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 16:11:57.622944 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:11:57.625231 5373 heartbeater.cc:347] Connected to a master server at 127.1.253.126:43035
I20260924 16:11:57.625497 5373 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:57.625998 5373 heartbeater.cc:511] Master 127.1.253.126:43035 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
I20260924 16:11:57.627679 5176 ts_manager.cc:194] Registered new tserver with Master: 767fd347190249d082350207d6aa44bb (127.1.253.65:42585)
I20260924 16:11:57.629499 5176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:50291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:11:57.643175 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43035
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:57.759581 5378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:57.759856 5378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:57.759899 5378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:57.763981 5378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:57.764048 5378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:57.764142 5378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:11:57.769737 5378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:57.770941 5378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:57.772080 5378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:57.782392 5386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:57.782459 5383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:57.782615 5384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:57.782641 5378 server_base.cc:1034] running on GCE node
I20260924 16:11:57.783267 5378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:57.783944 5378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:57.785141 5378 hybrid_clock.cc:648] HybridClock initialized: now 1790266317785119 us; error 34 us; skew 500 ppm
Sep 24 16:11:57 dist-test-slave-6wvm krb5kdc[5131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266317, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:57.788944 5378 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:11:57.790026 5378 webserver.cc:460] Webserver started at http://127.1.253.66:43265/ using document root <none> and password file <none>
I20260924 16:11:57.790774 5378 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:57.790867 5378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:57.791100 5378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:57.793239 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "379243a5a3774507abda8b35e17743ae"
format_stamp: "Formatted at 2026-09-24 16:11:57 on dist-test-slave-6wvm"
server_key: "e7f022c426e33a61829e92f6f2d61eb1"
server_key_iv: "2b0b7b1a5a8d423874323332268cbb9f"
server_key_version: "encryptionkey@0"
I20260924 16:11:57.793853 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "379243a5a3774507abda8b35e17743ae"
format_stamp: "Formatted at 2026-09-24 16:11:57 on dist-test-slave-6wvm"
server_key: "e7f022c426e33a61829e92f6f2d61eb1"
server_key_iv: "2b0b7b1a5a8d423874323332268cbb9f"
server_key_version: "encryptionkey@0"
I20260924 16:11:57.797816 5378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:11:57.800500 5393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:57.801877 5378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:11:57.802029 5378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "379243a5a3774507abda8b35e17743ae"
format_stamp: "Formatted at 2026-09-24 16:11:57 on dist-test-slave-6wvm"
server_key: "e7f022c426e33a61829e92f6f2d61eb1"
server_key_iv: "2b0b7b1a5a8d423874323332268cbb9f"
server_key_version: "encryptionkey@0"
I20260924 16:11:57.802148 5378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:57.817024 5378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:57.822793 5378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:57.823014 5378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:57.823664 5378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:57.824551 5378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:57.824622 5378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:57.824694 5378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:57.824746 5378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:57.834143 5378 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:35047
I20260924 16:11:57.834162 5507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:35047 every 8 connection(s)
I20260924 16:11:57.835142 5378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:11:57.840255 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 5378
I20260924 16:11:57.840415 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:11:57.840766 2037 external_mini_cluster.cc:1428] Setting key cdda08ee0cc9104ba8b4b8dcd8fc349b
Sep 24 16:11:57 dist-test-slave-6wvm krb5kdc[5131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266317, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:11:57.854398 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:57.837118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:36223 (local address 127.1.253.126:43035)
0924 16:11:57.837306 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:57.837311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:57.838182 (+ 871us) server_negotiation.cc:407] Connection header received
0924 16:11:57.839829 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:57.839832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:57.839880 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:57.839986 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:57.841803 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.842570 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.844455 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.844680 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.844770 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:57.848197 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:57.848204 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:57.848210 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:57.848248 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:57.851409 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:57.852129 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:57.852135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:57.852137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:57.852243 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:57.852740 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:57.852742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:57.852744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:57.853115 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:57.853195 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:57.854046 (+ 851us) server_negotiation.cc:299] Negotiation successful
0924 16:11:57.854178 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
I20260924 16:11:57.855556 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:57.837508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:43035 (local address 127.1.253.66:36223)
0924 16:11:57.838007 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:57.838056 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:57.839606 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:57.840268 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:57.840282 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:57.840867 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:57.841628 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:57.841650 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.842718 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.842722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:57.844259 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:57.844274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:57.844848 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:57.844854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:57.845096 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:57.846398 (+ 1302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:57.846426 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:57.848012 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:57.851565 (+ 3553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:57.851572 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:57.851588 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:57.852019 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:57.852414 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:57.852417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:57.852419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:57.852627 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:57.853234 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:57.853243 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:57.853727 (+ 484us) client_negotiation.cc:769] Sending connection context
0924 16:11:57.853951 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 16:11:57.854314 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
I20260924 16:11:57.856870 5508 heartbeater.cc:347] Connected to a master server at 127.1.253.126:43035
I20260924 16:11:57.857163 5508 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:57.857712 5508 heartbeater.cc:511] Master 127.1.253.126:43035 requested a full tablet report, sending...
I20260924 16:11:57.858978 5176 ts_manager.cc:194] Registered new tserver with Master: 379243a5a3774507abda8b35e17743ae (127.1.253.66:35047)
I20260924 16:11:57.859965 5176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:36223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:11:57.872946 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43035
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:57.989578 5513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:57.989821 5513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:57.989864 5513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:57.993848 5513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:57.993906 5513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:57.993963 5513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:11:57.999187 5513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:58.000241 5513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:58.001289 5513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:58.011271 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:58.011375 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:58.011328 5513 server_base.cc:1034] running on GCE node
W20260924 16:11:58.011348 5519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:58.012002 5513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:58.012786 5513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:58.013955 5513 hybrid_clock.cc:648] HybridClock initialized: now 1790266318013924 us; error 46 us; skew 500 ppm
Sep 24 16:11:58 dist-test-slave-6wvm krb5kdc[5131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:58.017660 5513 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:11:58.018620 5513 webserver.cc:460] Webserver started at http://127.1.253.67:39077/ using document root <none> and password file <none>
I20260924 16:11:58.019146 5513 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:58.019203 5513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:58.019416 5513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:58.021486 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "1eb1a2891d094458bba3d23095e8377c"
format_stamp: "Formatted at 2026-09-24 16:11:58 on dist-test-slave-6wvm"
server_key: "48f2bbce5c20a8e11d7b588a6a034548"
server_key_iv: "34056c4f4f11ee795f06340b81dee3a4"
server_key_version: "encryptionkey@0"
I20260924 16:11:58.021984 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "1eb1a2891d094458bba3d23095e8377c"
format_stamp: "Formatted at 2026-09-24 16:11:58 on dist-test-slave-6wvm"
server_key: "48f2bbce5c20a8e11d7b588a6a034548"
server_key_iv: "34056c4f4f11ee795f06340b81dee3a4"
server_key_version: "encryptionkey@0"
I20260924 16:11:58.025701 5513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:11:58.028160 5528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:58.029439 5513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:11:58.029583 5513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "1eb1a2891d094458bba3d23095e8377c"
format_stamp: "Formatted at 2026-09-24 16:11:58 on dist-test-slave-6wvm"
server_key: "48f2bbce5c20a8e11d7b588a6a034548"
server_key_iv: "34056c4f4f11ee795f06340b81dee3a4"
server_key_version: "encryptionkey@0"
I20260924 16:11:58.029695 5513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:58.053068 5513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:58.059041 5513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:58.059260 5513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:58.060068 5513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:58.060966 5513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:58.061038 5513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:58.061103 5513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:58.061153 5513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:58.069978 5513 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:35235
I20260924 16:11:58.070000 5642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:35235 every 8 connection(s)
I20260924 16:11:58.070941 5513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:11:58.080574 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 5513
I20260924 16:11:58.080724 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:11:58.081040 2037 external_mini_cluster.cc:1428] Setting key 62d891e4760a82cb375172a040296f62
Sep 24 16:11:58 dist-test-slave-6wvm krb5kdc[5131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:58.088608 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:58.072878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:51105 (local address 127.1.253.126:43035)
0924 16:11:58.073069 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:58.073074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:58.073918 (+ 844us) server_negotiation.cc:407] Connection header received
0924 16:11:58.075469 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:58.075472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:58.075520 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:58.075597 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:58.076898 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:58.077437 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:58.078636 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:58.078850 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:58.078935 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:58.082446 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:58.082453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:58.082460 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:58.082512 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:58.086039 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:58.086730 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:58.086734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:58.086735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:58.086839 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:58.087255 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:58.087258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:58.087260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:58.087569 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:58.087671 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:58.088296 (+ 625us) server_negotiation.cc:299] Negotiation successful
0924 16:11:58.088423 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:11:58.089702 5645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:58.073306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:43035 (local address 127.1.253.67:51105)
0924 16:11:58.073759 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:58.073802 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:58.075246 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:58.075757 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:58.075763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:58.076252 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:58.076755 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:58.076769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:58.077556 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:58.077558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:58.078525 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:58.078534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:58.079010 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:58.079013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:58.079151 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:58.080257 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:58.080274 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:58.082149 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:58.086190 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:58.086200 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:58.086217 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:58.086621 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:58.086932 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:58.086953 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:58.086955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:58.087139 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:58.087670 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:58.087695 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:58.088009 (+ 314us) client_negotiation.cc:769] Sending connection context
0924 16:11:58.088236 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 16:11:58.088575 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":117,"threads_started":1}
I20260924 16:11:58.090903 5643 heartbeater.cc:347] Connected to a master server at 127.1.253.126:43035
I20260924 16:11:58.091167 5643 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:58.091665 5643 heartbeater.cc:511] Master 127.1.253.126:43035 requested a full tablet report, sending...
I20260924 16:11:58.092713 5176 ts_manager.cc:194] Registered new tserver with Master: 1eb1a2891d094458bba3d23095e8377c (127.1.253.67:35235)
I20260924 16:11:58.093283 5176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:51105
I20260924 16:11:58.095599 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:11:58 dist-test-slave-6wvm krb5kdc[5131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266318, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 16:11:58 dist-test-slave-6wvm krb5kdc[5131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266318, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:58.122982 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:58.112690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47438 (local address 127.1.253.126:43035)
0924 16:11:58.112848 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:58.112853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:58.113021 (+ 168us) server_negotiation.cc:407] Connection header received
0924 16:11:58.113127 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:58.113130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:58.113175 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:58.113246 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:58.113912 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:58.114483 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:58.115288 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:58.115520 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:58.115600 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:58.118486 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:58.118489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:58.118491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:58.118514 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:58.121331 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:58.121717 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:58.121720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:58.121721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:58.121799 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:58.122055 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:58.122058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:58.122059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:58.122366 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:58.122499 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:58.122691 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 16:11:58.122782 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 16:11:58.123440 5176 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47438
I20260924 16:11:58.124346 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 5240
I20260924 16:11:58.131603 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 5378
I20260924 16:11:58.137766 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 5513
I20260924 16:11:58.143926 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 5144
2026-09-24T16:11:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2211 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:11:58 dist-test-slave-6wvm krb5kdc[5655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:11:58 dist-test-slave-6wvm krb5kdc[5655](info): set up 2 sockets
Sep 24 16:11:58 dist-test-slave-6wvm krb5kdc[5655](info): commencing operation
krb5kdc: starting...
W20260924 16:11:59.210004 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:11:59 dist-test-slave-6wvm krb5kdc[5655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:11:59.331625 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:34501
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:34501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:59.443737 5668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:59.443982 5668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:59.444021 5668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:59.447942 5668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:11:59.447999 5668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:59.448017 5668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:59.448033 5668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:11:59.448048 5668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:11:59.453176 5668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:34501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:34501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:59.454188 5668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:59.455286 5668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:59.463780 5674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:59.463758 5676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:59.463799 5673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:59.463908 5668 server_base.cc:1034] running on GCE node
I20260924 16:11:59.464566 5668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:59.465634 5668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:59.466804 5668 hybrid_clock.cc:648] HybridClock initialized: now 1790266319466784 us; error 29 us; skew 500 ppm
Sep 24 16:11:59 dist-test-slave-6wvm krb5kdc[5655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266319, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:59.470029 5668 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:11:59.470978 5668 webserver.cc:460] Webserver started at http://127.1.253.126:33619/ using document root <none> and password file <none>
I20260924 16:11:59.471688 5668 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:59.471752 5668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:59.471995 5668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:59.474030 5668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "0c8291d903f544ce849a2b94f7f55ed8"
format_stamp: "Formatted at 2026-09-24 16:11:59 on dist-test-slave-6wvm"
server_key: "00e0e2e1b1e6a36ad1a11393a5129a43"
server_key_iv: "a27549847c478501604dd584d3eef5b9"
server_key_version: "encryptionkey@0"
I20260924 16:11:59.474658 5668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "0c8291d903f544ce849a2b94f7f55ed8"
format_stamp: "Formatted at 2026-09-24 16:11:59 on dist-test-slave-6wvm"
server_key: "00e0e2e1b1e6a36ad1a11393a5129a43"
server_key_iv: "a27549847c478501604dd584d3eef5b9"
server_key_version: "encryptionkey@0"
I20260924 16:11:59.478364 5668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:11:59.480835 5683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:59.482059 5668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:11:59.482192 5668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "0c8291d903f544ce849a2b94f7f55ed8"
format_stamp: "Formatted at 2026-09-24 16:11:59 on dist-test-slave-6wvm"
server_key: "00e0e2e1b1e6a36ad1a11393a5129a43"
server_key_iv: "a27549847c478501604dd584d3eef5b9"
server_key_version: "encryptionkey@0"
I20260924 16:11:59.482365 5668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:59.508085 5668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:59.512696 5668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:59.512921 5668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:59.520488 5668 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:34501
I20260924 16:11:59.520787 5736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:34501 every 8 connection(s)
I20260924 16:11:59.521543 5668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:11:59.524377 5737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:11:59.528532 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 5668
I20260924 16:11:59.528661 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:11:59.528944 2037 external_mini_cluster.cc:1428] Setting key 2acac8cb9bcc8940fb8b39b98f38b069
I20260924 16:11:59.530622 5737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8: Bootstrap starting.
I20260924 16:11:59.532799 5737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8: Neither blocks nor log segments found. Creating new log.
I20260924 16:11:59.533520 5737 log.cc:824] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8: Log is configured to *not* fsync() on all Append() calls
I20260924 16:11:59.535452 5737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8: No bootstrap required, opened a new log
Sep 24 16:11:59 dist-test-slave-6wvm krb5kdc[5655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:11:59.539913 5737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8291d903f544ce849a2b94f7f55ed8" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34501 } }
I20260924 16:11:59.540218 5737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:11:59.540304 5737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8291d903f544ce849a2b94f7f55ed8, State: Initialized, Role: FOLLOWER
I20260924 16:11:59.540844 5737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8291d903f544ce849a2b94f7f55ed8" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34501 } }
I20260924 16:11:59.541008 5737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:11:59.541100 5737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:11:59.541232 5737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:11:59.542347 5737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8291d903f544ce849a2b94f7f55ed8" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34501 } }
I20260924 16:11:59.542698 5737 leader_election.cc:302] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c8291d903f544ce849a2b94f7f55ed8; no voters:
I20260924 16:11:59.543047 5737 leader_election.cc:288] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:11:59.543176 5742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:11:59.543399 5742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8291d903f544ce849a2b94f7f55ed8, State: Running, Role: LEADER
I20260924 16:11:59.543717 5742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8291d903f544ce849a2b94f7f55ed8" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34501 } }
I20260924 16:11:59.544421 5737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:11:59.545282 5743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c8291d903f544ce849a2b94f7f55ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8291d903f544ce849a2b94f7f55ed8" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34501 } } }
I20260924 16:11:59.545399 5743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:59.545707 5744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c8291d903f544ce849a2b94f7f55ed8. Latest consensus state: current_term: 1 leader_uuid: "0c8291d903f544ce849a2b94f7f55ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8291d903f544ce849a2b94f7f55ed8" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34501 } } }
I20260924 16:11:59.545782 5744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8 [sys.catalog]: This master's current role is: LEADER
I20260924 16:11:59.546042 5751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:11:59.546577 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:59.530463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38382 (local address 127.1.253.126:34501)
0924 16:11:59.530925 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:59.530941 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:59.530977 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:11:59.531597 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:59.531613 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:59.531955 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:59.532336 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:59.533184 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:59.534393 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:59.535317 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:59.535867 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:59.536048 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:59.539554 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:59.539559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:59.539573 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:59.539600 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:59.542774 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:59.543374 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:59.543380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:59.543386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:59.543550 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:59.543865 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:59.543868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:59.543869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:59.544373 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:59.544561 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:59.545112 (+ 551us) server_negotiation.cc:299] Negotiation successful
0924 16:11:59.545470 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":125,"threads_started":1}
I20260924 16:11:59.549199 5751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:11:59.554692 5751 catalog_manager.cc:1347] Generated new cluster ID: 21d0e0c40e3949e5acf571def9acdded
I20260924 16:11:59.554760 5751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:11:59.595000 5751 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:11:59.596041 5751 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:11:59.611174 5751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c8291d903f544ce849a2b94f7f55ed8: Generated new TSK 0
I20260924 16:11:59.612042 5751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:11:59.641942 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:34501
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:59.760905 5763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:59.761227 5763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:59.761313 5763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:59.765630 5763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:59.765722 5763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:59.765820 5763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:11:59.771241 5763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:34501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:59.772367 5763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:59.773464 5763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:11:59.783360 5771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:59.783367 5768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:11:59.783367 5769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:11:59.783593 5763 server_base.cc:1034] running on GCE node
I20260924 16:11:59.784192 5763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:11:59.784871 5763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:11:59.786029 5763 hybrid_clock.cc:648] HybridClock initialized: now 1790266319785987 us; error 59 us; skew 500 ppm
Sep 24 16:11:59 dist-test-slave-6wvm krb5kdc[5655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266319, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:11:59.789534 5763 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:11:59.790581 5763 webserver.cc:460] Webserver started at http://127.1.253.65:34705/ using document root <none> and password file <none>
I20260924 16:11:59.791189 5763 fs_manager.cc:359] Metadata directory not provided
I20260924 16:11:59.791282 5763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:11:59.791541 5763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:11:59.793666 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "9958a4f84072413098969f5b7acb1249"
format_stamp: "Formatted at 2026-09-24 16:11:59 on dist-test-slave-6wvm"
server_key: "7353f32b0e008d8175db386f8791bd51"
server_key_iv: "548701d4739ce265e9a9a5fa30e49293"
server_key_version: "encryptionkey@0"
I20260924 16:11:59.794317 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "9958a4f84072413098969f5b7acb1249"
format_stamp: "Formatted at 2026-09-24 16:11:59 on dist-test-slave-6wvm"
server_key: "7353f32b0e008d8175db386f8791bd51"
server_key_iv: "548701d4739ce265e9a9a5fa30e49293"
server_key_version: "encryptionkey@0"
I20260924 16:11:59.798079 5763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:11:59.800793 5778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:59.802059 5763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:11:59.802208 5763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "9958a4f84072413098969f5b7acb1249"
format_stamp: "Formatted at 2026-09-24 16:11:59 on dist-test-slave-6wvm"
server_key: "7353f32b0e008d8175db386f8791bd51"
server_key_iv: "548701d4739ce265e9a9a5fa30e49293"
server_key_version: "encryptionkey@0"
I20260924 16:11:59.802373 5763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:11:59.819329 5763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:11:59.823818 5763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:11:59.824033 5763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:11:59.824659 5763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:11:59.825536 5763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:11:59.825609 5763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:59.825678 5763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:11:59.825722 5763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:11:59.835021 5763 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:37959
I20260924 16:11:59.835044 5891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:37959 every 8 connection(s)
I20260924 16:11:59.835935 5763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:11:59.839311 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 5763
I20260924 16:11:59.839448 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:11:59.839852 2037 external_mini_cluster.cc:1428] Setting key 5979d901242aa7ab5ff11245adbb977b
Sep 24 16:11:59 dist-test-slave-6wvm krb5kdc[5655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266319, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
I20260924 16:11:59.853387 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:59.837877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:34877 (local address 127.1.253.126:34501)
0924 16:11:59.838071 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 16:11:59.838076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:11:59.838983 (+ 907us) server_negotiation.cc:407] Connection header received
0924 16:11:59.840807 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:11:59.840810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:11:59.840860 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:11:59.840950 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:11:59.842573 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:59.843137 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:59.844353 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:11:59.844593 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:11:59.844707 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:59.848393 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:11:59.848399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:11:59.848402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:11:59.848440 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:11:59.851173 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:59.851716 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:59.851719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:59.851720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:59.851799 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:11:59.852124 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:11:59.852126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:11:59.852127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:11:59.852306 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:11:59.852379 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 16:11:59.852970 (+ 591us) server_negotiation.cc:299] Negotiation successful
0924 16:11:59.853125 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 16:11:59.855273 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:11:59.838372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:34501 (local address 127.1.253.65:34877)
0924 16:11:59.838826 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 16:11:59.838871 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:11:59.840559 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:11:59.841126 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:11:59.841134 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:11:59.841657 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:11:59.842414 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:59.842439 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:59.843280 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:59.843283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:59.844226 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:11:59.844236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:11:59.844753 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:11:59.844757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:11:59.844981 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:11:59.846476 (+ 1495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:11:59.846493 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:11:59.848216 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:11:59.851282 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:59.851287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:59.851300 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:59.851555 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:59.851879 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:11:59.851882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:11:59.851883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:11:59.852034 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:11:59.852393 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:11:59.852398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:11:59.852745 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 16:11:59.852906 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 16:11:59.853283 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260924 16:11:59.856609 5892 heartbeater.cc:347] Connected to a master server at 127.1.253.126:34501
I20260924 16:11:59.856892 5892 heartbeater.cc:464] Registering TS with master...
I20260924 16:11:59.857443 5892 heartbeater.cc:511] Master 127.1.253.126:34501 requested a full tablet report, sending...
I20260924 16:11:59.859099 5697 ts_manager.cc:194] Registered new tserver with Master: 9958a4f84072413098969f5b7acb1249 (127.1.253.65:37959)
I20260924 16:11:59.860355 5697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:34877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:11:59.868947 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:34501
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:11:59.982139 5897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:11:59.982442 5897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:11:59.982519 5897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:11:59.986719 5897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:11:59.986771 5897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:11:59.986873 5897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:11:59.992123 5897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:34501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:11:59.993194 5897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:11:59.994228 5897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:00.004333 5902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:00.004333 5903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:00.004377 5906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:00.004825 5897 server_base.cc:1034] running on GCE node
I20260924 16:12:00.005301 5897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:00.006031 5897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:00.007221 5897 hybrid_clock.cc:648] HybridClock initialized: now 1790266320007177 us; error 66 us; skew 500 ppm
Sep 24 16:12:00 dist-test-slave-6wvm krb5kdc[5655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266320, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:00.010629 5897 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:12:00.011677 5897 webserver.cc:460] Webserver started at http://127.1.253.66:35115/ using document root <none> and password file <none>
I20260924 16:12:00.012298 5897 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:00.012394 5897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:00.012655 5897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:00.014853 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "8e667600045744afbb1ceaa31d045538"
format_stamp: "Formatted at 2026-09-24 16:12:00 on dist-test-slave-6wvm"
server_key: "995513defe15a18dbe9bee5b446867c4"
server_key_iv: "1b5d611963b5efbc448dd051a6c38532"
server_key_version: "encryptionkey@0"
I20260924 16:12:00.015481 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "8e667600045744afbb1ceaa31d045538"
format_stamp: "Formatted at 2026-09-24 16:12:00 on dist-test-slave-6wvm"
server_key: "995513defe15a18dbe9bee5b446867c4"
server_key_iv: "1b5d611963b5efbc448dd051a6c38532"
server_key_version: "encryptionkey@0"
I20260924 16:12:00.019425 5897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:12:00.022095 5913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:00.023389 5897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:00.023542 5897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "8e667600045744afbb1ceaa31d045538"
format_stamp: "Formatted at 2026-09-24 16:12:00 on dist-test-slave-6wvm"
server_key: "995513defe15a18dbe9bee5b446867c4"
server_key_iv: "1b5d611963b5efbc448dd051a6c38532"
server_key_version: "encryptionkey@0"
I20260924 16:12:00.023658 5897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:00.038203 5897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:00.042528 5897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:00.042730 5897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:00.043340 5897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:00.044193 5897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:00.044263 5897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:00.044337 5897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:00.044379 5897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:00.053735 5897 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:36765
I20260924 16:12:00.053752 6027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:36765 every 8 connection(s)
I20260924 16:12:00.054741 5897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:00.055739 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 5897
I20260924 16:12:00.055850 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:00.056160 2037 external_mini_cluster.cc:1428] Setting key b37f39f4d43f8ba794b1c4716e424dee
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
Sep 24 16:12:00 dist-test-slave-6wvm krb5kdc[5655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266320, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:00.075132 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:00.056814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:50341 (local address 127.1.253.126:34501)
0924 16:12:00.056975 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:00.056980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:00.058013 (+ 1033us) server_negotiation.cc:407] Connection header received
0924 16:12:00.060036 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:00.060041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:00.060107 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:00.060222 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:00.061518 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:00.062453 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:00.063608 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:00.064046 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:00.064182 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:00.070024 (+ 5842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:00.070028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:00.070031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:00.070067 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:00.072958 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:00.073497 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:00.073501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:00.073502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:00.073592 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:00.073945 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:00.073948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:00.073950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:00.074155 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:00.074300 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:00.074824 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 16:12:00.074952 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:12:00.076102 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:00.057269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:34501 (local address 127.1.253.66:50341)
0924 16:12:00.057846 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:00.057903 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:00.059782 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:00.060386 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:00.060393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:00.060850 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:00.061365 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:00.061379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:00.062582 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:00.062585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:00.063477 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:00.063487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:00.064209 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:00.064212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:00.064538 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:00.065913 (+ 1375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:00.065930 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:00.069851 (+ 3921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:00.073089 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:00.073095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:00.073113 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:00.073384 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:00.073692 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:00.073695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:00.073698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:00.073844 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:00.074317 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:00.074324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:00.074580 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 16:12:00.074785 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 16:12:00.075117 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":165,"threads_started":1}
I20260924 16:12:00.077410 6028 heartbeater.cc:347] Connected to a master server at 127.1.253.126:34501
I20260924 16:12:00.077697 6028 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:00.078349 6028 heartbeater.cc:511] Master 127.1.253.126:34501 requested a full tablet report, sending...
I20260924 16:12:00.079561 5697 ts_manager.cc:194] Registered new tserver with Master: 8e667600045744afbb1ceaa31d045538 (127.1.253.66:36765)
I20260924 16:12:00.080207 5697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:50341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:12:00.085325 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:34501
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:00.203118 6033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:00.203380 6033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:00.203434 6033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:00.207605 6033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:00.207670 6033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:00.207731 6033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:12:00.212908 6033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:34501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:00.213898 6033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:00.214994 6033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:00.224905 6038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:00.224905 6042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:00.224932 6039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:00.225039 6033 server_base.cc:1034] running on GCE node
I20260924 16:12:00.225638 6033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:00.226359 6033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:00.227557 6033 hybrid_clock.cc:648] HybridClock initialized: now 1790266320227519 us; error 51 us; skew 500 ppm
Sep 24 16:12:00 dist-test-slave-6wvm krb5kdc[5655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266320, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:00.231055 6033 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:12:00.231971 6033 webserver.cc:460] Webserver started at http://127.1.253.67:45179/ using document root <none> and password file <none>
I20260924 16:12:00.232564 6033 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:00.232659 6033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:00.232918 6033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:00.235140 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "4897c7b4668f4396866d4f31bd8358e3"
format_stamp: "Formatted at 2026-09-24 16:12:00 on dist-test-slave-6wvm"
server_key: "9106db5d875e495791c1c5a685ba5fc5"
server_key_iv: "715af68770adf926d33834c88b2f56ae"
server_key_version: "encryptionkey@0"
I20260924 16:12:00.235747 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "4897c7b4668f4396866d4f31bd8358e3"
format_stamp: "Formatted at 2026-09-24 16:12:00 on dist-test-slave-6wvm"
server_key: "9106db5d875e495791c1c5a685ba5fc5"
server_key_iv: "715af68770adf926d33834c88b2f56ae"
server_key_version: "encryptionkey@0"
I20260924 16:12:00.239665 6033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:12:00.242484 6049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:00.243712 6033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:00.243860 6033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "4897c7b4668f4396866d4f31bd8358e3"
format_stamp: "Formatted at 2026-09-24 16:12:00 on dist-test-slave-6wvm"
server_key: "9106db5d875e495791c1c5a685ba5fc5"
server_key_iv: "715af68770adf926d33834c88b2f56ae"
server_key_version: "encryptionkey@0"
I20260924 16:12:00.244029 6033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:00.255821 6033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:00.260577 6033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:00.260807 6033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:00.261468 6033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:00.262485 6033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:00.262573 6033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:00.262672 6033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:00.262727 6033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:00.272732 6033 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:43751
I20260924 16:12:00.272761 6164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:43751 every 8 connection(s)
I20260924 16:12:00.273718 6033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:00.283371 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 6033
I20260924 16:12:00.283519 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:00.283846 2037 external_mini_cluster.cc:1428] Setting key bb2cf177ad74637dbbebef8caf9075ef
Sep 24 16:12:00 dist-test-slave-6wvm krb5kdc[5655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266320, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:00.290990 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:00.275732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:48941 (local address 127.1.253.126:34501)
0924 16:12:00.275881 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:00.275885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:00.276784 (+ 899us) server_negotiation.cc:407] Connection header received
0924 16:12:00.278201 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:00.278204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:00.278311 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:00.278412 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:00.279653 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:00.280214 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:00.281396 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:00.281626 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:00.281749 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:00.285432 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:00.285437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:00.285440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:00.285469 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:00.288565 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:00.289125 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:00.289130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:00.289132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:00.289228 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:00.289648 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:00.289653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:00.289655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:00.289865 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:00.289994 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:00.290632 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 16:12:00.290790 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:12:00.291844 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:00.276129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:34501 (local address 127.1.253.67:48941)
0924 16:12:00.276622 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:00.276669 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:00.277968 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:00.278628 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:00.278635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:00.278986 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:00.279499 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:00.279513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:00.280354 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:00.280358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:00.281259 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:00.281269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:00.281790 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:00.281794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:00.281979 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:00.283452 (+ 1473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:00.283480 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:00.285224 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:00.288709 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:00.288716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:00.288733 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:00.289011 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:00.289358 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:00.289363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:00.289365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:00.289537 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:00.289994 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:00.290003 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:00.290344 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 16:12:00.290573 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 16:12:00.290947 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":110,"threads_started":1}
I20260924 16:12:00.293028 6165 heartbeater.cc:347] Connected to a master server at 127.1.253.126:34501
I20260924 16:12:00.293298 6165 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:00.293807 6165 heartbeater.cc:511] Master 127.1.253.126:34501 requested a full tablet report, sending...
I20260924 16:12:00.294890 5697 ts_manager.cc:194] Registered new tserver with Master: 4897c7b4668f4396866d4f31bd8358e3 (127.1.253.67:43751)
I20260924 16:12:00.295450 5697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:48941
I20260924 16:12:00.297926 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:12:00 dist-test-slave-6wvm krb5kdc[5655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:12:00 dist-test-slave-6wvm krb5kdc[5655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.65@KRBTEST.COM
I20260924 16:12:00.326683 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:00.315383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.1.253.65:37959)
0924 16:12:00.315694 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:00.315712 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:00.315743 (+ 31us) server_negotiation.cc:407] Connection header received
0924 16:12:00.315817 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:00.315822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:00.316026 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:00.316183 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:00.316894 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:00.317671 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:00.318608 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:00.318992 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:00.319123 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:00.322232 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:00.322272 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:00.322295 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:00.322329 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:00.324753 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:00.325203 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:00.325214 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:00.325216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:00.325294 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:00.325579 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:00.325583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:00.325585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:00.325835 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:00.326115 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:00.326302 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 16:12:00.326470 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
Sep 24 16:12:00 dist-test-slave-6wvm krb5kdc[5655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:00.340329 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:00.329652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.1.253.126:34501)
0924 16:12:00.329791 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:00.329798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:00.329879 (+ 81us) server_negotiation.cc:407] Connection header received
0924 16:12:00.330039 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:00.330042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:00.330097 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:00.330184 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:00.330846 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:00.331364 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:00.332448 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:00.332837 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:00.332955 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:00.336310 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:00.336313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:00.336315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:00.336341 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:00.338642 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:00.339040 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:00.339043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:00.339044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:00.339131 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:00.339380 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:00.339382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:00.339383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:00.339633 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:00.339748 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:00.339962 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 16:12:00.340069 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 16:12:00.340812 5697 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38392
I20260924 16:12:00.341547 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 5763
I20260924 16:12:00.348984 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 5897
I20260924 16:12:00.354784 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 6033
I20260924 16:12:00.360857 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 5668
2026-09-24T16:12:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2216 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:00 dist-test-slave-6wvm krb5kdc[6177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:00 dist-test-slave-6wvm krb5kdc[6177](info): set up 2 sockets
Sep 24 16:12:00 dist-test-slave-6wvm krb5kdc[6177](info): commencing operation
krb5kdc: starting...
W20260924 16:12:01.403005 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:12:01 dist-test-slave-6wvm krb5kdc[6177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:12:01.529973 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:46413
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:46413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:01.644490 6190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:01.644750 6190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:01.644795 6190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:01.649325 6190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:01.649389 6190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:01.649459 6190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:01.649504 6190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:01.649561 6190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:01.655071 6190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:46413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:46413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:01.656219 6190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:01.657348 6190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:01.665999 6196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:01.666056 6195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:01.666181 6190 server_base.cc:1034] running on GCE node
W20260924 16:12:01.666056 6198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:01.666898 6190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:01.668018 6190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:01.669207 6190 hybrid_clock.cc:648] HybridClock initialized: now 1790266321669174 us; error 53 us; skew 500 ppm
Sep 24 16:12:01 dist-test-slave-6wvm krb5kdc[6177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:01.681634 6190 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:12:01.682683 6190 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:01.682807 6190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:01.683089 6190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:01.685375 6190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "5925e76ca3734b7c8d37c3aacf6aec4a"
format_stamp: "Formatted at 2026-09-24 16:12:01 on dist-test-slave-6wvm"
server_key: "8609fb08b777d4e0f23244e220cb2ced"
server_key_iv: "41e0c473ae2a67701be3c7762950a640"
server_key_version: "encryptionkey@0"
I20260924 16:12:01.686018 6190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "5925e76ca3734b7c8d37c3aacf6aec4a"
format_stamp: "Formatted at 2026-09-24 16:12:01 on dist-test-slave-6wvm"
server_key: "8609fb08b777d4e0f23244e220cb2ced"
server_key_iv: "41e0c473ae2a67701be3c7762950a640"
server_key_version: "encryptionkey@0"
I20260924 16:12:01.690027 6190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:12:01.692770 6204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:01.694060 6190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:01.694280 6190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "5925e76ca3734b7c8d37c3aacf6aec4a"
format_stamp: "Formatted at 2026-09-24 16:12:01 on dist-test-slave-6wvm"
server_key: "8609fb08b777d4e0f23244e220cb2ced"
server_key_iv: "41e0c473ae2a67701be3c7762950a640"
server_key_version: "encryptionkey@0"
I20260924 16:12:01.694409 6190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:01.711611 6190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:01.715102 6190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:01.715322 6190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:01.722579 6190 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:46413
I20260924 16:12:01.722863 6257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:46413 every 8 connection(s)
I20260924 16:12:01.723311 6190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:01.726058 6258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:01.726980 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 6190
I20260924 16:12:01.727121 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:01.727437 2037 external_mini_cluster.cc:1428] Setting key ac23d1229d5dfecad8186ec80ae106c7
I20260924 16:12:01.732381 6258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a: Bootstrap starting.
I20260924 16:12:01.734627 6258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:01.735342 6258 log.cc:824] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:12:01 dist-test-slave-6wvm krb5kdc[6177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:01.737280 6258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a: No bootstrap required, opened a new log
I20260924 16:12:01.741079 6258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5925e76ca3734b7c8d37c3aacf6aec4a" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46413 } }
I20260924 16:12:01.741314 6258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:01.741393 6258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5925e76ca3734b7c8d37c3aacf6aec4a, State: Initialized, Role: FOLLOWER
I20260924 16:12:01.741580 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:01.728925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.1.253.126:46413)
0924 16:12:01.729353 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:01.729367 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:01.729403 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:12:01.730044 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:01.730060 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:01.730414 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:01.730766 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:01.731573 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:01.732560 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:01.733433 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:01.733924 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:01.734095 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:01.736786 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:01.736792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:01.736804 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:01.736830 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:01.738688 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:01.739139 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:01.739144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:01.739148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:01.739253 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:01.739632 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:01.739635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:01.739636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:01.739991 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:01.740161 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:01.740465 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 16:12:01.740777 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":136,"threads_started":1}
I20260924 16:12:01.741899 6258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5925e76ca3734b7c8d37c3aacf6aec4a" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46413 } }
I20260924 16:12:01.742071 6258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:01.742139 6258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:01.742261 6258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:01.743278 6258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5925e76ca3734b7c8d37c3aacf6aec4a" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46413 } }
I20260924 16:12:01.743628 6258 leader_election.cc:302] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5925e76ca3734b7c8d37c3aacf6aec4a; no voters:
I20260924 16:12:01.743985 6258 leader_election.cc:288] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:01.744041 6263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:01.744222 6263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [term 1 LEADER]: Becoming Leader. State: Replica: 5925e76ca3734b7c8d37c3aacf6aec4a, State: Running, Role: LEADER
I20260924 16:12:01.744529 6263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5925e76ca3734b7c8d37c3aacf6aec4a" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46413 } }
I20260924 16:12:01.745199 6258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:01.746299 6265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5925e76ca3734b7c8d37c3aacf6aec4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5925e76ca3734b7c8d37c3aacf6aec4a" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46413 } } }
I20260924 16:12:01.746424 6265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:01.746767 6264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5925e76ca3734b7c8d37c3aacf6aec4a. Latest consensus state: current_term: 1 leader_uuid: "5925e76ca3734b7c8d37c3aacf6aec4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5925e76ca3734b7c8d37c3aacf6aec4a" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46413 } } }
I20260924 16:12:01.746860 6264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:01.746945 6272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:01.750031 6272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:01.755738 6272 catalog_manager.cc:1347] Generated new cluster ID: 7a0b8b02c90e45bf80668e45434799d8
I20260924 16:12:01.755802 6272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:01.762583 6272 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:01.763736 6272 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:01.776403 6272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a: Generated new TSK 0
I20260924 16:12:01.777109 6272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:12:01.818554 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:46413
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:01.937043 6285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:01.937335 6285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:01.937413 6285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:01.941501 6285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:01.941588 6285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:01.941680 6285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:12:01.947166 6285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:46413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:01.948195 6285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:01.949239 6285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:01.958630 6290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:01.958709 6285 server_base.cc:1034] running on GCE node
W20260924 16:12:01.958652 6293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:01.958637 6291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:01.959214 6285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:01.959875 6285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:01.961079 6285 hybrid_clock.cc:648] HybridClock initialized: now 1790266321961055 us; error 35 us; skew 500 ppm
Sep 24 16:12:01 dist-test-slave-6wvm krb5kdc[6177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:01.963935 6285 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:12:01.964761 6285 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:01.964865 6285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:01.965128 6285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:01.967649 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "3bdecf7199d84217a316f272f8303dc6"
format_stamp: "Formatted at 2026-09-24 16:12:01 on dist-test-slave-6wvm"
server_key: "1250d57d4baaafdc2f8cdd1c572ee9ff"
server_key_iv: "1def2e278fa7103330d98c9e55490a76"
server_key_version: "encryptionkey@0"
I20260924 16:12:01.968261 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "3bdecf7199d84217a316f272f8303dc6"
format_stamp: "Formatted at 2026-09-24 16:12:01 on dist-test-slave-6wvm"
server_key: "1250d57d4baaafdc2f8cdd1c572ee9ff"
server_key_iv: "1def2e278fa7103330d98c9e55490a76"
server_key_version: "encryptionkey@0"
I20260924 16:12:01.972358 6285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:12:01.974890 6300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:01.976177 6285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:01.976315 6285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "3bdecf7199d84217a316f272f8303dc6"
format_stamp: "Formatted at 2026-09-24 16:12:01 on dist-test-slave-6wvm"
server_key: "1250d57d4baaafdc2f8cdd1c572ee9ff"
server_key_iv: "1def2e278fa7103330d98c9e55490a76"
server_key_version: "encryptionkey@0"
I20260924 16:12:01.976435 6285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:01.994508 6285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:01.997355 6285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:01.997552 6285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:01.998134 6285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:01.999022 6285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:01.999091 6285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:01.999163 6285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:01.999207 6285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:02.008232 6285 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:33655
I20260924 16:12:02.008252 6414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:33655 every 8 connection(s)
I20260924 16:12:02.008821 6285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:02.016238 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 6285
I20260924 16:12:02.016403 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:02.016702 2037 external_mini_cluster.cc:1428] Setting key 387aff57618085f605a6f7367d04c3d5
Sep 24 16:12:02 dist-test-slave-6wvm krb5kdc[6177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:02.023437 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.010764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:41941 (local address 127.1.253.126:46413)
0924 16:12:02.010912 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:02.010916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:02.011827 (+ 911us) server_negotiation.cc:407] Connection header received
0924 16:12:02.013233 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:02.013237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:02.013298 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:02.013413 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:02.014713 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.015259 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.016414 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.016720 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.016831 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.019598 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:02.019602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:02.019604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:02.019642 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:02.021186 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:02.021717 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:02.021725 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:02.021727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:02.021805 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:02.022206 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:02.022209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:02.022211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:02.022516 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:02.022625 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:02.023135 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 16:12:02.023265 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:12:02.024307 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.011187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:46413 (local address 127.1.253.65:41941)
0924 16:12:02.011672 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:02.011716 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:02.012999 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:02.013599 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:02.013607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:02.013969 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:02.014562 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.014578 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.015394 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.015403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.016281 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.016291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.016890 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.016894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.017065 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.017704 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:02.017722 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:02.019427 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:02.021320 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:02.021325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:02.021342 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:02.021604 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:02.021952 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:02.021960 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:02.021962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:02.022096 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:02.022625 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:02.022632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:02.022893 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 16:12:02.023093 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 16:12:02.023442 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
I20260924 16:12:02.025724 6415 heartbeater.cc:347] Connected to a master server at 127.1.253.126:46413
I20260924 16:12:02.025978 6415 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:12:02.026507 6415 heartbeater.cc:511] Master 127.1.253.126:46413 requested a full tablet report, sending...
I20260924 16:12:02.028048 6222 ts_manager.cc:194] Registered new tserver with Master: 3bdecf7199d84217a316f272f8303dc6 (127.1.253.65:33655)
I20260924 16:12:02.029315 6222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:41941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:12:02.045138 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:46413
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:02.161460 6420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:02.161734 6420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:02.161808 6420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:02.165727 6420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:02.165805 6420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:02.165892 6420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:12:02.171271 6420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:46413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:02.172260 6420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:02.173264 6420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:02.183058 6425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:02.183223 6420 server_base.cc:1034] running on GCE node
W20260924 16:12:02.183076 6426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:02.183107 6428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:02.183790 6420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:02.184473 6420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:02.185648 6420 hybrid_clock.cc:648] HybridClock initialized: now 1790266322185627 us; error 30 us; skew 500 ppm
Sep 24 16:12:02 dist-test-slave-6wvm krb5kdc[6177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266322, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:02.188757 6420 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:12:02.189589 6420 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:02.189658 6420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:02.189935 6420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:02.192152 6420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "46295e8559cf469fa6b6abd3f8eab789"
format_stamp: "Formatted at 2026-09-24 16:12:02 on dist-test-slave-6wvm"
server_key: "599e14ebc965baa6f68691242943addd"
server_key_iv: "ccef157a5259e94570da82afd1649f9d"
server_key_version: "encryptionkey@0"
I20260924 16:12:02.192791 6420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "46295e8559cf469fa6b6abd3f8eab789"
format_stamp: "Formatted at 2026-09-24 16:12:02 on dist-test-slave-6wvm"
server_key: "599e14ebc965baa6f68691242943addd"
server_key_iv: "ccef157a5259e94570da82afd1649f9d"
server_key_version: "encryptionkey@0"
I20260924 16:12:02.196645 6420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:12:02.199244 6434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:02.200482 6420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:02.200629 6420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "46295e8559cf469fa6b6abd3f8eab789"
format_stamp: "Formatted at 2026-09-24 16:12:02 on dist-test-slave-6wvm"
server_key: "599e14ebc965baa6f68691242943addd"
server_key_iv: "ccef157a5259e94570da82afd1649f9d"
server_key_version: "encryptionkey@0"
I20260924 16:12:02.200742 6420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:02.223831 6420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:02.226910 6420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:02.227068 6420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:02.227634 6420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:02.228449 6420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:02.228497 6420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:02.228547 6420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:02.228570 6420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:02.237751 6420 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:45813
I20260924 16:12:02.237769 6549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:45813 every 8 connection(s)
I20260924 16:12:02.238432 6420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:02.241428 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 6420
I20260924 16:12:02.241569 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:02.241894 2037 external_mini_cluster.cc:1428] Setting key 73b43ec1e34f908cdcacbb0e036987f7
Sep 24 16:12:02 dist-test-slave-6wvm krb5kdc[6177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266322, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:12:02.254384 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.240342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:41909 (local address 127.1.253.126:46413)
0924 16:12:02.240487 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:02.240491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:02.241483 (+ 992us) server_negotiation.cc:407] Connection header received
0924 16:12:02.243166 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:02.243169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:02.243213 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:02.243287 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:02.244558 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.245432 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.247315 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.247607 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.247689 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.250578 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:02.250584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:02.250588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:02.250619 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:02.252263 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:02.252789 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:02.252794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:02.252798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:02.252896 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:02.253283 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:02.253286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:02.253287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:02.253468 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:02.253558 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:02.254077 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 16:12:02.254182 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:12:02.255333 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.240767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:46413 (local address 127.1.253.66:41909)
0924 16:12:02.241294 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:02.241355 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:02.242947 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:02.243525 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:02.243534 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:02.243921 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:02.244409 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.244422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.245557 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.245564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.247197 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.247212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.247762 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.247769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.248009 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.248674 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:02.248703 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:02.250386 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:02.252384 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:02.252389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:02.252400 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:02.252669 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:02.253005 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:02.253011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:02.253016 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:02.253160 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:02.253594 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:02.253599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:02.253832 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 16:12:02.253985 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 16:12:02.254360 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
I20260924 16:12:02.256709 6550 heartbeater.cc:347] Connected to a master server at 127.1.253.126:46413
I20260924 16:12:02.256975 6550 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:02.257469 6550 heartbeater.cc:511] Master 127.1.253.126:46413 requested a full tablet report, sending...
I20260924 16:12:02.258599 6222 ts_manager.cc:194] Registered new tserver with Master: 46295e8559cf469fa6b6abd3f8eab789 (127.1.253.66:45813)
I20260924 16:12:02.259186 6222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:41909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:12:02.271234 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:46413
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:02.386109 6555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:02.386430 6555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:02.386479 6555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:02.390398 6555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:02.390450 6555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:02.390507 6555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:12:02.395642 6555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:46413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:02.396646 6555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:02.397686 6555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:02.407524 6560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:02.407433 6563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:02.407439 6561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:02.407799 6555 server_base.cc:1034] running on GCE node
I20260924 16:12:02.408177 6555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:02.408793 6555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:02.409945 6555 hybrid_clock.cc:648] HybridClock initialized: now 1790266322409917 us; error 49 us; skew 500 ppm
Sep 24 16:12:02 dist-test-slave-6wvm krb5kdc[6177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266322, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:02.412726 6555 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:12:02.413455 6555 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:02.413516 6555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:02.413745 6555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:02.415829 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "1529184eee3c4289ade99fb84c5e7607"
format_stamp: "Formatted at 2026-09-24 16:12:02 on dist-test-slave-6wvm"
server_key: "2a9a0b6178f57a2b46d9c8dcb1d65e99"
server_key_iv: "2459b8554ed5a27d53a20a1e594adf31"
server_key_version: "encryptionkey@0"
I20260924 16:12:02.416392 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "1529184eee3c4289ade99fb84c5e7607"
format_stamp: "Formatted at 2026-09-24 16:12:02 on dist-test-slave-6wvm"
server_key: "2a9a0b6178f57a2b46d9c8dcb1d65e99"
server_key_iv: "2459b8554ed5a27d53a20a1e594adf31"
server_key_version: "encryptionkey@0"
I20260924 16:12:02.420118 6555 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:12:02.422539 6569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:02.423712 6555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:12:02.423846 6555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "1529184eee3c4289ade99fb84c5e7607"
format_stamp: "Formatted at 2026-09-24 16:12:02 on dist-test-slave-6wvm"
server_key: "2a9a0b6178f57a2b46d9c8dcb1d65e99"
server_key_iv: "2459b8554ed5a27d53a20a1e594adf31"
server_key_version: "encryptionkey@0"
I20260924 16:12:02.423952 6555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:02.441565 6555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:02.445050 6555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:02.445247 6555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:02.445945 6555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:02.446995 6555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:02.447072 6555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:02.447146 6555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:02.447184 6555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:02.456513 6555 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:33157
I20260924 16:12:02.456547 6683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:33157 every 8 connection(s)
I20260924 16:12:02.457180 6555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:02.457509 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 6555
I20260924 16:12:02.457651 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:02.458009 2037 external_mini_cluster.cc:1428] Setting key 00b0214b52df50016cf3e2f69bfc74b3
Sep 24 16:12:02 dist-test-slave-6wvm krb5kdc[6177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266322, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:02.471803 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.459303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:46373 (local address 127.1.253.126:46413)
0924 16:12:02.459470 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:02.459475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:02.460584 (+ 1109us) server_negotiation.cc:407] Connection header received
0924 16:12:02.461950 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:02.461953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:02.461999 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:02.462067 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:02.463299 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.463855 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.465013 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.465227 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.465302 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.468024 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:02.468028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:02.468030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:02.468059 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:02.469422 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:02.470079 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:02.470082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:02.470083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:02.470171 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:02.470565 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:02.470568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:02.470569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:02.470760 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:02.470876 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:02.471462 (+ 586us) server_negotiation.cc:299] Negotiation successful
0924 16:12:02.471628 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:12:02.472774 6686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.459856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:46413 (local address 127.1.253.67:46373)
0924 16:12:02.460383 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:02.460479 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:02.461754 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:02.462209 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:02.462216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:02.462664 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:02.463162 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.463177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.463993 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.463996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.464901 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.464912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.465362 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.465365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.465541 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.466282 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:02.466316 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:02.467822 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:02.469547 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:02.469552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:02.469564 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:02.469974 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:02.470306 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:02.470311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:02.470315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:02.470468 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:02.470878 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:02.470886 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:02.471138 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 16:12:02.471359 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 16:12:02.471723 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":106,"threads_started":1}
I20260924 16:12:02.473821 6684 heartbeater.cc:347] Connected to a master server at 127.1.253.126:46413
I20260924 16:12:02.474042 6684 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:02.474582 6684 heartbeater.cc:511] Master 127.1.253.126:46413 requested a full tablet report, sending...
I20260924 16:12:02.475586 6222 ts_manager.cc:194] Registered new tserver with Master: 1529184eee3c4289ade99fb84c5e7607 (127.1.253.67:33157)
I20260924 16:12:02.476121 6222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:46373
I20260924 16:12:02.484378 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:12:02.493448 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.486358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.1.253.126:46413)
0924 16:12:02.486494 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:02.486498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:02.486724 (+ 226us) server_negotiation.cc:407] Connection header received
0924 16:12:02.486827 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:02.486829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:02.486871 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:02.486939 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:02.487647 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.488149 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.488936 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.489157 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.489230 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.490596 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:02.490599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:02.490601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:02.490624 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:02.492057 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:02.492416 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:02.492418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:02.492419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:02.492492 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:02.492713 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:02.492715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:02.492716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:02.492920 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:02.493017 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:02.493225 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 16:12:02.493326 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:12:02.496230 6222 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:02.499493 6222 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:02.514793 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.508299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:33655 (local address 127.0.0.1:49406)
0924 16:12:02.509816 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:02.509831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:02.509936 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:02.510456 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:02.510458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:02.510476 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:02.510578 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.510584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.511731 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.511736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.513030 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.513041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.514142 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.514146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.514289 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.514355 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 16:12:02.514471 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:12:02.514604 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1454,"mutex_wait_us":82,"thread_start_us":139,"threads_started":1}
I20260924 16:12:02.514825 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.509090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.1.253.65:33655)
0924 16:12:02.509679 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:02.509686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:02.509943 (+ 257us) server_negotiation.cc:407] Connection header received
0924 16:12:02.510048 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:02.510053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:02.510286 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:02.510487 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:02.510677 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.511529 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.513168 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.514000 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.514116 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.514312 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:02.514524 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 16:12:02.514636 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":111,"threads_started":1}
I20260924 16:12:02.514994 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.508997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:33157 (local address 127.0.0.1:43176)
0924 16:12:02.510117 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:02.510140 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:02.510222 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:02.510745 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:02.510747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:02.510765 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:02.510861 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.510867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.512404 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.512408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.513502 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.513509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.514588 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.514592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.514694 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.514716 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:12:02.514786 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 16:12:02.514869 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"thread_start_us":93,"threads_started":1}
I20260924 16:12:02.515494 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.508207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:45813 (local address 127.0.0.1:49268)
0924 16:12:02.508872 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:02.508926 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:02.509057 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:02.510044 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:02.510051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:02.510080 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:02.510228 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.510268 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.511755 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.511759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.513275 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.513291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.514556 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.514560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.514647 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.514670 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:12:02.514744 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 16:12:02.514816 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":49,"thread_start_us":90,"threads_started":1}
I20260924 16:12:02.515491 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.509481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.1.253.67:33157)
0924 16:12:02.509796 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:02.509815 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:02.510352 (+ 537us) server_negotiation.cc:407] Connection header received
0924 16:12:02.510439 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:02.510444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:02.510621 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:02.510775 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:02.510990 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.512249 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.513617 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.514244 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.514374 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.514529 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:02.515223 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 16:12:02.515333 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":121,"threads_started":1}
I20260924 16:12:02.516462 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.508304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.1.253.66:45813)
0924 16:12:02.509364 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:02.509378 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:02.509403 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:12:02.509488 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:02.509494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:02.509872 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:02.510047 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:02.510373 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.511582 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.513437 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.514201 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.514326 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.514490 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:02.516212 (+ 1722us) server_negotiation.cc:299] Negotiation successful
0924 16:12:02.516304 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":130,"threads_started":1}
I20260924 16:12:02.518452 6618 tablet_service.cc:1467] Processing CreateTablet for tablet 7aeddae413e74542a12be61b86eec310 (DEFAULT_TABLE table=test-table [id=f3404165f2d94a2aa5a1d5ff06fc1bdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:02.518458 6349 tablet_service.cc:1467] Processing CreateTablet for tablet 7aeddae413e74542a12be61b86eec310 (DEFAULT_TABLE table=test-table [id=f3404165f2d94a2aa5a1d5ff06fc1bdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:02.519001 6483 tablet_service.cc:1467] Processing CreateTablet for tablet 7aeddae413e74542a12be61b86eec310 (DEFAULT_TABLE table=test-table [id=f3404165f2d94a2aa5a1d5ff06fc1bdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:02.519613 6349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aeddae413e74542a12be61b86eec310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:02.519623 6618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aeddae413e74542a12be61b86eec310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:02.519914 6483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aeddae413e74542a12be61b86eec310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:02.526645 6701 tablet_bootstrap.cc:492] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607: Bootstrap starting.
I20260924 16:12:02.526651 6702 tablet_bootstrap.cc:492] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6: Bootstrap starting.
I20260924 16:12:02.527724 6703 tablet_bootstrap.cc:492] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789: Bootstrap starting.
I20260924 16:12:02.529098 6702 tablet_bootstrap.cc:654] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:02.529098 6701 tablet_bootstrap.cc:654] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:02.529827 6703 tablet_bootstrap.cc:654] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:02.530100 6702 log.cc:824] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:02.530505 6701 log.cc:824] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:02.531116 6703 log.cc:824] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:02.532600 6701 tablet_bootstrap.cc:492] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607: No bootstrap required, opened a new log
I20260924 16:12:02.532600 6702 tablet_bootstrap.cc:492] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6: No bootstrap required, opened a new log
I20260924 16:12:02.532802 6703 tablet_bootstrap.cc:492] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789: No bootstrap required, opened a new log
I20260924 16:12:02.532799 6701 ts_tablet_manager.cc:1397] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:12:02.532799 6702 ts_tablet_manager.cc:1397] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:12:02.533010 6703 ts_tablet_manager.cc:1397] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:12:02.535794 6703 raft_consensus.cc:348] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46295e8559cf469fa6b6abd3f8eab789" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45813 } } peers { permanent_uuid: "3bdecf7199d84217a316f272f8303dc6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33655 } } peers { permanent_uuid: "1529184eee3c4289ade99fb84c5e7607" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33157 } }
I20260924 16:12:02.536053 6703 raft_consensus.cc:374] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:02.536108 6703 raft_consensus.cc:729] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46295e8559cf469fa6b6abd3f8eab789, State: Initialized, Role: FOLLOWER
I20260924 16:12:02.536574 6703 consensus_queue.cc:260] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46295e8559cf469fa6b6abd3f8eab789" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45813 } } peers { permanent_uuid: "3bdecf7199d84217a316f272f8303dc6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33655 } } peers { permanent_uuid: "1529184eee3c4289ade99fb84c5e7607" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33157 } }
I20260924 16:12:02.536751 6701 raft_consensus.cc:348] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46295e8559cf469fa6b6abd3f8eab789" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45813 } } peers { permanent_uuid: "3bdecf7199d84217a316f272f8303dc6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33655 } } peers { permanent_uuid: "1529184eee3c4289ade99fb84c5e7607" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33157 } }
I20260924 16:12:02.537020 6701 raft_consensus.cc:374] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:02.537068 6701 raft_consensus.cc:729] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1529184eee3c4289ade99fb84c5e7607, State: Initialized, Role: FOLLOWER
I20260924 16:12:02.537137 6702 raft_consensus.cc:348] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46295e8559cf469fa6b6abd3f8eab789" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45813 } } peers { permanent_uuid: "3bdecf7199d84217a316f272f8303dc6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33655 } } peers { permanent_uuid: "1529184eee3c4289ade99fb84c5e7607" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33157 } }
I20260924 16:12:02.537353 6702 raft_consensus.cc:374] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:02.537400 6702 raft_consensus.cc:729] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bdecf7199d84217a316f272f8303dc6, State: Initialized, Role: FOLLOWER
I20260924 16:12:02.537544 6701 consensus_queue.cc:260] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46295e8559cf469fa6b6abd3f8eab789" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45813 } } peers { permanent_uuid: "3bdecf7199d84217a316f272f8303dc6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33655 } } peers { permanent_uuid: "1529184eee3c4289ade99fb84c5e7607" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33157 } }
I20260924 16:12:02.537690 6703 ts_tablet_manager.cc:1428] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:12:02.537796 6702 consensus_queue.cc:260] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46295e8559cf469fa6b6abd3f8eab789" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45813 } } peers { permanent_uuid: "3bdecf7199d84217a316f272f8303dc6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33655 } } peers { permanent_uuid: "1529184eee3c4289ade99fb84c5e7607" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33157 } }
I20260924 16:12:02.538085 6550 heartbeater.cc:503] Master 127.1.253.126:46413 was elected leader, sending a full tablet report...
I20260924 16:12:02.538544 6701 ts_tablet_manager.cc:1428] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 16:12:02.538648 6702 ts_tablet_manager.cc:1428] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 16:12:02.538892 6684 heartbeater.cc:503] Master 127.1.253.126:46413 was elected leader, sending a full tablet report...
I20260924 16:12:02.538934 6415 heartbeater.cc:503] Master 127.1.253.126:46413 was elected leader, sending a full tablet report...
W20260924 16:12:02.708367 6685 tablet.cc:2369] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:02.721690 6709 raft_consensus.cc:482] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:02.721939 6709 raft_consensus.cc:504] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46295e8559cf469fa6b6abd3f8eab789" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45813 } } peers { permanent_uuid: "3bdecf7199d84217a316f272f8303dc6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33655 } } peers { permanent_uuid: "1529184eee3c4289ade99fb84c5e7607" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33157 } }
I20260924 16:12:02.723300 6709 leader_election.cc:288] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46295e8559cf469fa6b6abd3f8eab789 (127.1.253.66:45813), 1529184eee3c4289ade99fb84c5e7607 (127.1.253.67:33157)
I20260924 16:12:02.728041 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.724042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:39855 (local address 127.1.253.67:33157)
0924 16:12:02.724166 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:02.724171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:02.724186 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:12:02.724274 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:02.724278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:02.724329 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:02.724427 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:02.724738 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.725540 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.726802 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.727435 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.727560 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.727613 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:02.727832 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 16:12:02.727914 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:12:02.728251 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.723546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:45813 (local address 127.1.253.65:33171)
0924 16:12:02.723973 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:02.723992 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:02.724173 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:02.724557 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:02.724560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:02.724574 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:02.724697 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.724702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.725670 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.725673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.726907 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.726919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.727802 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.727806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.727917 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.727935 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 16:12:02.728025 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 16:12:02.728103 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":125,"threads_started":1}
I20260924 16:12:02.728268 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.723646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:33171 (local address 127.1.253.66:45813)
0924 16:12:02.723801 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:02.723806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:02.724070 (+ 264us) server_negotiation.cc:407] Connection header received
0924 16:12:02.724328 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:02.724331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:02.724376 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:02.724450 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:02.724799 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.725540 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.727020 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.727615 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.727765 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.727815 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:02.728066 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 16:12:02.728165 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:12:02.728884 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.723606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:33157 (local address 127.1.253.65:39855)
0924 16:12:02.724070 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:02.724084 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:02.724183 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:02.724449 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:02.724453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:02.724469 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:02.724610 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.724616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.725712 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.725715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.726667 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:02.726680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.727570 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.727574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:02.727677 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.727704 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:12:02.728504 (+ 800us) client_negotiation.cc:240] Negotiation successful
0924 16:12:02.728592 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260924 16:12:02.729029 6503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aeddae413e74542a12be61b86eec310" candidate_uuid: "3bdecf7199d84217a316f272f8303dc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46295e8559cf469fa6b6abd3f8eab789" is_pre_election: true
I20260924 16:12:02.729370 6503 raft_consensus.cc:2454] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bdecf7199d84217a316f272f8303dc6 in term 0.
I20260924 16:12:02.729744 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aeddae413e74542a12be61b86eec310" candidate_uuid: "3bdecf7199d84217a316f272f8303dc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1529184eee3c4289ade99fb84c5e7607" is_pre_election: true
I20260924 16:12:02.729929 6303 leader_election.cc:302] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bdecf7199d84217a316f272f8303dc6, 46295e8559cf469fa6b6abd3f8eab789; no voters:
I20260924 16:12:02.730093 6638 raft_consensus.cc:2454] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bdecf7199d84217a316f272f8303dc6 in term 0.
I20260924 16:12:02.730293 6709 raft_consensus.cc:2781] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:02.730370 6709 raft_consensus.cc:482] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:02.730408 6709 raft_consensus.cc:3037] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:02.731681 6709 raft_consensus.cc:504] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46295e8559cf469fa6b6abd3f8eab789" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45813 } } peers { permanent_uuid: "3bdecf7199d84217a316f272f8303dc6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33655 } } peers { permanent_uuid: "1529184eee3c4289ade99fb84c5e7607" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33157 } }
I20260924 16:12:02.732057 6709 leader_election.cc:288] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [CANDIDATE]: Term 1 election: Requested vote from peers 46295e8559cf469fa6b6abd3f8eab789 (127.1.253.66:45813), 1529184eee3c4289ade99fb84c5e7607 (127.1.253.67:33157)
I20260924 16:12:02.732430 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aeddae413e74542a12be61b86eec310" candidate_uuid: "3bdecf7199d84217a316f272f8303dc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1529184eee3c4289ade99fb84c5e7607"
I20260924 16:12:02.732447 6503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aeddae413e74542a12be61b86eec310" candidate_uuid: "3bdecf7199d84217a316f272f8303dc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46295e8559cf469fa6b6abd3f8eab789"
I20260924 16:12:02.732551 6638 raft_consensus.cc:3037] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:02.732596 6503 raft_consensus.cc:3037] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:02.733711 6503 raft_consensus.cc:2454] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bdecf7199d84217a316f272f8303dc6 in term 1.
I20260924 16:12:02.733796 6638 raft_consensus.cc:2454] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bdecf7199d84217a316f272f8303dc6 in term 1.
I20260924 16:12:02.734119 6303 leader_election.cc:302] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bdecf7199d84217a316f272f8303dc6, 46295e8559cf469fa6b6abd3f8eab789; no voters:
I20260924 16:12:02.734359 6709 raft_consensus.cc:2781] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:02.734774 6709 raft_consensus.cc:686] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [term 1 LEADER]: Becoming Leader. State: Replica: 3bdecf7199d84217a316f272f8303dc6, State: Running, Role: LEADER
I20260924 16:12:02.735172 6709 consensus_queue.cc:237] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46295e8559cf469fa6b6abd3f8eab789" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45813 } } peers { permanent_uuid: "3bdecf7199d84217a316f272f8303dc6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33655 } } peers { permanent_uuid: "1529184eee3c4289ade99fb84c5e7607" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33157 } }
I20260924 16:12:02.738577 6221 catalog_manager.cc:5495] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bdecf7199d84217a316f272f8303dc6 (127.1.253.65). New cstate: current_term: 1 leader_uuid: "3bdecf7199d84217a316f272f8303dc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46295e8559cf469fa6b6abd3f8eab789" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bdecf7199d84217a316f272f8303dc6" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1529184eee3c4289ade99fb84c5e7607" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33157 } health_report { overall_health: UNKNOWN } } }
W20260924 16:12:02.739532 6551 tablet.cc:2369] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:02.750627 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.747002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.1.253.65:33655)
0924 16:12:02.747138 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:02.747142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:02.747156 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:12:02.747390 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:02.747393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:02.747449 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:02.747539 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:02.747780 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.748271 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.749148 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:02.749349 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:02.749423 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:02.749722 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:02.750149 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:02.750300 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:02.750377 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 16:12:02.750428 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:12:02.756911 6638 raft_consensus.cc:1261] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607 [term 1 FOLLOWER]: Refusing update from remote peer 3bdecf7199d84217a316f272f8303dc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:02.756911 6503 raft_consensus.cc:1261] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789 [term 1 FOLLOWER]: Refusing update from remote peer 3bdecf7199d84217a316f272f8303dc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:02.757705 6712 consensus_queue.cc:1035] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46295e8559cf469fa6b6abd3f8eab789" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:02.757887 6709 consensus_queue.cc:1035] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1529184eee3c4289ade99fb84c5e7607" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:12:02.760185 6416 tablet.cc:2369] T 7aeddae413e74542a12be61b86eec310 P 3bdecf7199d84217a316f272f8303dc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:02.768872 6717 mvcc.cc:205] Tried to move back new op lower bound from 7332930858004312064 to 7332930857926266880. Current Snapshot: MvccSnapshot[applied={T|T < 7332930858004312064}]
I20260924 16:12:02.774827 6221 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:02.774981 6221 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:02.777341 6221 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5925e76ca3734b7c8d37c3aacf6aec4a: Sending DeleteTablet for 3 replicas of tablet 7aeddae413e74542a12be61b86eec310
I20260924 16:12:02.778153 6349 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aeddae413e74542a12be61b86eec310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:02 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:49406
I20260924 16:12:02.778486 6483 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aeddae413e74542a12be61b86eec310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:02 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:49268
I20260924 16:12:02.778910 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 6285
I20260924 16:12:02.779026 6726 tablet_replica.cc:330] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789: stopping tablet replica
I20260924 16:12:02.779026 6618 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aeddae413e74542a12be61b86eec310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:02 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:43176
I20260924 16:12:02.779547 6726 raft_consensus.cc:2229] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:02.779927 6726 raft_consensus.cc:2258] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:02.781687 6726 ts_tablet_manager.cc:1905] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:02.781968 6727 tablet_replica.cc:330] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607: stopping tablet replica
I20260924 16:12:02.782322 6727 raft_consensus.cc:2229] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:02.782617 6727 raft_consensus.cc:2258] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:02.784276 6726 ts_tablet_manager.cc:1918] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:02.784273 6727 ts_tablet_manager.cc:1905] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:02.784354 6726 log.cc:1196] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/7aeddae413e74542a12be61b86eec310
I20260924 16:12:02.784684 6726 ts_tablet_manager.cc:1939] T 7aeddae413e74542a12be61b86eec310 P 46295e8559cf469fa6b6abd3f8eab789: Deleting consensus metadata
I20260924 16:12:02.785770 6208 catalog_manager.cc:4841] TS 46295e8559cf469fa6b6abd3f8eab789 (127.1.253.66:45813): tablet 7aeddae413e74542a12be61b86eec310 (table test-table [id=f3404165f2d94a2aa5a1d5ff06fc1bdd]) successfully deleted
I20260924 16:12:02.787220 6727 ts_tablet_manager.cc:1918] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:02.787322 6727 log.cc:1196] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/7aeddae413e74542a12be61b86eec310
I20260924 16:12:02.787684 6727 ts_tablet_manager.cc:1939] T 7aeddae413e74542a12be61b86eec310 P 1529184eee3c4289ade99fb84c5e7607: Deleting consensus metadata
W20260924 16:12:02.788044 6206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:33655 (error 108)
I20260924 16:12:02.788944 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 6420
I20260924 16:12:02.789048 6209 catalog_manager.cc:4841] TS 1529184eee3c4289ade99fb84c5e7607 (127.1.253.67:33157): tablet 7aeddae413e74542a12be61b86eec310 (table test-table [id=f3404165f2d94a2aa5a1d5ff06fc1bdd]) successfully deleted
I20260924 16:12:02.789224 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:02.788789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:33655 (local address 127.0.0.1:49422)
0924 16:12:02.789025 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:02.789122 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:33655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260924 16:12:02.789903 6206 catalog_manager.cc:4584] TS 3bdecf7199d84217a316f272f8303dc6 (127.1.253.65:33655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7aeddae413e74542a12be61b86eec310: Network error: Client connection negotiation failed: client connection to 127.1.253.65:33655: connect: Connection refused (error 111)
I20260924 16:12:02.796886 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 6555
I20260924 16:12:02.802842 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 6190
2026-09-24T16:12:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2441 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T16:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:02Z Disabled control of system clock
I20260924 16:12:02.833802 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:39317
--webserver_interface=127.1.253.126
--webserver_port=0
--builtin_ntp_servers=127.1.253.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:39317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 16:12:02.945973 6732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:02.946228 6732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:02.946333 6732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:02.950209 6732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:02.950306 6732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:02.950327 6732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:02.950344 6732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:02.950358 6732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:02.954985 6732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:39317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:39317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:02.955984 6732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:02.957369 6732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:02.965989 6738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:02.966096 6732 server_base.cc:1034] running on GCE node
W20260924 16:12:02.966146 6737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:02.965963 6741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:02.966792 6732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:02.967940 6732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:02.969138 6732 hybrid_clock.cc:648] HybridClock initialized: now 1790266322969096 us; error 52 us; skew 500 ppm
I20260924 16:12:02.971158 6732 webserver.cc:460] Webserver started at http://127.1.253.126:42611/ using document root <none> and password file <none>
I20260924 16:12:02.971745 6732 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:02.971835 6732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:02.972069 6732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:02.974155 6732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "c757dacd100445fbba0999e87674c5a5"
format_stamp: "Formatted at 2026-09-24 16:12:02 on dist-test-slave-6wvm"
server_key: "f1f7bbe833f56b492d8eb00d40e3e3e1"
server_key_iv: "e3f4735c43838094f1bfecfb288c46d0"
server_key_version: "encryptionkey@0"
I20260924 16:12:02.974774 6732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "c757dacd100445fbba0999e87674c5a5"
format_stamp: "Formatted at 2026-09-24 16:12:02 on dist-test-slave-6wvm"
server_key: "f1f7bbe833f56b492d8eb00d40e3e3e1"
server_key_iv: "e3f4735c43838094f1bfecfb288c46d0"
server_key_version: "encryptionkey@0"
I20260924 16:12:02.978499 6732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:02.981009 6747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:02.982288 6732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:02.982432 6732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "c757dacd100445fbba0999e87674c5a5"
format_stamp: "Formatted at 2026-09-24 16:12:02 on dist-test-slave-6wvm"
server_key: "f1f7bbe833f56b492d8eb00d40e3e3e1"
server_key_iv: "e3f4735c43838094f1bfecfb288c46d0"
server_key_version: "encryptionkey@0"
I20260924 16:12:02.982542 6732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:02.987257 6732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:02.987862 6732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:02.988046 6732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:02.995433 6732 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:39317
I20260924 16:12:02.995445 6799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:39317 every 8 connection(s)
I20260924 16:12:02.996418 6732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:02.999434 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 6732
I20260924 16:12:02.999575 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:02.999871 2037 external_mini_cluster.cc:1428] Setting key dbdd91c219df416307a49a276ac9c9cb
I20260924 16:12:03.001087 6800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:03.005590 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.001392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.1.253.126:39317)
0924 16:12:03.002041 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:03.002055 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:03.002090 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:12:03.002824 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:03.002842 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:03.003204 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:03.003558 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:03.004363 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:03.004368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:03.004380 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:03.004390 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:03.004444 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:03.004459 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:03.004616 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:03.004692 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 16:12:03.004928 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":133,"threads_started":1}
I20260924 16:12:03.007483 6800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5: Bootstrap starting.
I20260924 16:12:03.009789 6800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:03.010660 6800 log.cc:824] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:03.012596 6800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5: No bootstrap required, opened a new log
I20260924 16:12:03.016084 6800 raft_consensus.cc:348] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c757dacd100445fbba0999e87674c5a5" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 39317 } }
I20260924 16:12:03.016355 6800 raft_consensus.cc:374] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:03.016392 6800 raft_consensus.cc:729] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c757dacd100445fbba0999e87674c5a5, State: Initialized, Role: FOLLOWER
I20260924 16:12:03.016775 6800 consensus_queue.cc:260] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c757dacd100445fbba0999e87674c5a5" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 39317 } }
I20260924 16:12:03.016894 6800 raft_consensus.cc:388] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:03.016940 6800 raft_consensus.cc:482] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:03.017011 6800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:03.018164 6800 raft_consensus.cc:504] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c757dacd100445fbba0999e87674c5a5" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 39317 } }
I20260924 16:12:03.018533 6800 leader_election.cc:302] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c757dacd100445fbba0999e87674c5a5; no voters:
I20260924 16:12:03.018880 6800 leader_election.cc:288] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:03.018991 6805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:03.019243 6805 raft_consensus.cc:686] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [term 1 LEADER]: Becoming Leader. State: Replica: c757dacd100445fbba0999e87674c5a5, State: Running, Role: LEADER
I20260924 16:12:03.019541 6805 consensus_queue.cc:237] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c757dacd100445fbba0999e87674c5a5" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 39317 } }
I20260924 16:12:03.020172 6800 sys_catalog.cc:564] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:03.021188 6806 sys_catalog.cc:455] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c757dacd100445fbba0999e87674c5a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c757dacd100445fbba0999e87674c5a5" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 39317 } } }
I20260924 16:12:03.021334 6806 sys_catalog.cc:458] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:03.021723 6814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:03.021687 6807 sys_catalog.cc:455] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c757dacd100445fbba0999e87674c5a5. Latest consensus state: current_term: 1 leader_uuid: "c757dacd100445fbba0999e87674c5a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c757dacd100445fbba0999e87674c5a5" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 39317 } } }
I20260924 16:12:03.022511 6807 sys_catalog.cc:458] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:03.024983 6814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:03.030565 6814 catalog_manager.cc:1347] Generated new cluster ID: 6f6892ac8eef476fa5f13232ff7b9c85
I20260924 16:12:03.030653 6814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:03.038754 6814 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:03.039064 6814 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:03.058195 6814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5: Generated new TSK 0
I20260924 16:12:03.059084 6814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:12:03.070892 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:39317
--builtin_ntp_servers=127.1.253.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 16:12:03.187157 6824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:03.187493 6824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:03.187539 6824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:03.191563 6824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:03.191628 6824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:03.191685 6824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:12:03.196421 6824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:39317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:03.197494 6824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:03.198906 6824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:03.208685 6830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:03.208663 6829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:03.208734 6824 server_base.cc:1034] running on GCE node
W20260924 16:12:03.208663 6832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:03.209388 6824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:03.209983 6824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:03.211148 6824 hybrid_clock.cc:648] HybridClock initialized: now 1790266323211117 us; error 36 us; skew 500 ppm
I20260924 16:12:03.213117 6824 webserver.cc:460] Webserver started at http://127.1.253.65:33283/ using document root <none> and password file <none>
I20260924 16:12:03.213737 6824 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:03.213802 6824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:03.214056 6824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:03.216255 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "8b79355e525e42fab3270608b3ebe88d"
format_stamp: "Formatted at 2026-09-24 16:12:03 on dist-test-slave-6wvm"
server_key: "8be4022db46b1a90369bf46739a1b449"
server_key_iv: "d21acdf16d17647b9d4657f3f8eada45"
server_key_version: "encryptionkey@0"
I20260924 16:12:03.216852 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "8b79355e525e42fab3270608b3ebe88d"
format_stamp: "Formatted at 2026-09-24 16:12:03 on dist-test-slave-6wvm"
server_key: "8be4022db46b1a90369bf46739a1b449"
server_key_iv: "d21acdf16d17647b9d4657f3f8eada45"
server_key_version: "encryptionkey@0"
I20260924 16:12:03.220834 6824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 16:12:03.223729 6838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:03.225075 6824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:12:03.225221 6824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "8b79355e525e42fab3270608b3ebe88d"
format_stamp: "Formatted at 2026-09-24 16:12:03 on dist-test-slave-6wvm"
server_key: "8be4022db46b1a90369bf46739a1b449"
server_key_iv: "d21acdf16d17647b9d4657f3f8eada45"
server_key_version: "encryptionkey@0"
I20260924 16:12:03.225340 6824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:03.230057 6824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:03.230763 6824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:03.230966 6824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:03.231536 6824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:03.232461 6824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:03.232542 6824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:03.232617 6824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:03.232671 6824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:03.241961 6824 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:42209
I20260924 16:12:03.242044 6951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:42209 every 8 connection(s)
I20260924 16:12:03.242959 6824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:03.247821 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 6824
I20260924 16:12:03.247951 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:03.248251 2037 external_mini_cluster.cc:1428] Setting key a1ce28079e4130ba1cb1de4d138b9e63
I20260924 16:12:03.249747 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.244975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:35873 (local address 127.1.253.126:39317)
0924 16:12:03.245152 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:03.245156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:03.246269 (+ 1113us) server_negotiation.cc:407] Connection header received
0924 16:12:03.247685 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:03.247689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:03.247740 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:03.247865 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:03.249227 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:03.249230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:03.249232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:03.249244 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:03.249264 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:03.249272 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:03.249363 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:03.249529 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 16:12:03.249567 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:12:03.250670 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:39317
--builtin_ntp_servers=127.1.253.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 16:12:03.250873 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.245388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:39317 (local address 127.1.253.65:35873)
0924 16:12:03.246079 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:03.246120 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:03.247428 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:03.248085 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:03.248096 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:03.248196 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:03.248908 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:03.248910 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:03.248951 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:03.249042 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:03.249371 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:03.249380 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:03.249386 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:12:03.249560 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 16:12:03.249840 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":105,"threads_started":1}
I20260924 16:12:03.252136 6952 heartbeater.cc:347] Connected to a master server at 127.1.253.126:39317
I20260924 16:12:03.252381 6952 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:03.252846 6952 heartbeater.cc:511] Master 127.1.253.126:39317 requested a full tablet report, sending...
I20260924 16:12:03.254568 6764 ts_manager.cc:194] Registered new tserver with Master: 8b79355e525e42fab3270608b3ebe88d (127.1.253.65:42209)
W20260924 16:12:03.368460 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:03.368760 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:03.368837 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:03.373082 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:03.373167 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:03.373258 6955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:12:03.378108 6955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:39317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:03.379303 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:03.380795 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:03.390385 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:03.390389 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:03.390398 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:03.390439 6955 server_base.cc:1034] running on GCE node
I20260924 16:12:03.391098 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:03.391659 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:03.392822 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790266323392802 us; error 30 us; skew 500 ppm
I20260924 16:12:03.394732 6955 webserver.cc:460] Webserver started at http://127.1.253.66:46605/ using document root <none> and password file <none>
I20260924 16:12:03.395258 6955 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:03.395313 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:03.395471 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:03.397426 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "44a8254909a54fee8abf0a4652bef11c"
format_stamp: "Formatted at 2026-09-24 16:12:03 on dist-test-slave-6wvm"
server_key: "0cad56d6434923a012dfb681557de215"
server_key_iv: "f7e29d29df053a7854886e4af1ed5380"
server_key_version: "encryptionkey@0"
I20260924 16:12:03.397939 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "44a8254909a54fee8abf0a4652bef11c"
format_stamp: "Formatted at 2026-09-24 16:12:03 on dist-test-slave-6wvm"
server_key: "0cad56d6434923a012dfb681557de215"
server_key_iv: "f7e29d29df053a7854886e4af1ed5380"
server_key_version: "encryptionkey@0"
I20260924 16:12:03.401469 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:12:03.403756 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:03.405236 6955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:12:03.405350 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "44a8254909a54fee8abf0a4652bef11c"
format_stamp: "Formatted at 2026-09-24 16:12:03 on dist-test-slave-6wvm"
server_key: "0cad56d6434923a012dfb681557de215"
server_key_iv: "f7e29d29df053a7854886e4af1ed5380"
server_key_version: "encryptionkey@0"
I20260924 16:12:03.405462 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:03.409884 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:03.410740 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:03.410928 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:03.411499 6955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:03.412390 6955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:03.412444 6955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:03.412484 6955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:03.412533 6955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:03.421707 6955 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:34937
I20260924 16:12:03.421741 7083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:34937 every 8 connection(s)
I20260924 16:12:03.422677 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:03.427881 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 6955
I20260924 16:12:03.427994 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:03.428411 2037 external_mini_cluster.cc:1428] Setting key 26877cfc6963098a38f59cab7f57c83f
I20260924 16:12:03.429405 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.424604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:60317 (local address 127.1.253.126:39317)
0924 16:12:03.424762 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:03.424766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:03.425884 (+ 1118us) server_negotiation.cc:407] Connection header received
0924 16:12:03.427345 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:03.427348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:03.427393 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:03.427484 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:03.428812 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:03.428817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:03.428822 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:03.428842 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:03.428871 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:03.428892 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:03.428997 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:03.429200 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 16:12:03.429253 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:12:03.430558 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.424925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:39317 (local address 127.1.253.66:60317)
0924 16:12:03.425702 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:03.425746 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:03.427102 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:03.427683 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:03.427690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:03.427786 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:03.428494 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:03.428498 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:03.428565 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:03.428641 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:03.429010 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:03.429022 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:03.429031 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:12:03.429225 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 16:12:03.429488 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":106,"threads_started":1}
I20260924 16:12:03.431774 7084 heartbeater.cc:347] Connected to a master server at 127.1.253.126:39317
I20260924 16:12:03.431962 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:39317
--builtin_ntp_servers=127.1.253.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 16:12:03.432044 7084 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:03.432516 7084 heartbeater.cc:511] Master 127.1.253.126:39317 requested a full tablet report, sending...
I20260924 16:12:03.433759 6764 ts_manager.cc:194] Registered new tserver with Master: 44a8254909a54fee8abf0a4652bef11c (127.1.253.66:34937)
W20260924 16:12:03.557049 7087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:03.557297 7087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:03.557335 7087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:03.561450 7087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:03.561501 7087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:03.561556 7087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:12:03.566390 7087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:39317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:03.567407 7087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:03.568843 7087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:03.578727 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:03.578727 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:03.578780 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:03.579087 7087 server_base.cc:1034] running on GCE node
I20260924 16:12:03.579553 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:03.580268 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:03.581457 7087 hybrid_clock.cc:648] HybridClock initialized: now 1790266323581445 us; error 36 us; skew 500 ppm
I20260924 16:12:03.583462 7087 webserver.cc:460] Webserver started at http://127.1.253.67:41953/ using document root <none> and password file <none>
I20260924 16:12:03.584084 7087 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:03.584188 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:03.584432 7087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:03.586910 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "6c4bafe8bbd54184ae6400104f88ba00"
format_stamp: "Formatted at 2026-09-24 16:12:03 on dist-test-slave-6wvm"
server_key: "0c760c53c107605e93441535035c4e09"
server_key_iv: "9958220e37a682f77bc71aa219fcd964"
server_key_version: "encryptionkey@0"
I20260924 16:12:03.587487 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "6c4bafe8bbd54184ae6400104f88ba00"
format_stamp: "Formatted at 2026-09-24 16:12:03 on dist-test-slave-6wvm"
server_key: "0c760c53c107605e93441535035c4e09"
server_key_iv: "9958220e37a682f77bc71aa219fcd964"
server_key_version: "encryptionkey@0"
I20260924 16:12:03.591842 7087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:12:03.594278 7101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:03.595677 7087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 16:12:03.595810 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "6c4bafe8bbd54184ae6400104f88ba00"
format_stamp: "Formatted at 2026-09-24 16:12:03 on dist-test-slave-6wvm"
server_key: "0c760c53c107605e93441535035c4e09"
server_key_iv: "9958220e37a682f77bc71aa219fcd964"
server_key_version: "encryptionkey@0"
I20260924 16:12:03.595930 7087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:03.600713 7087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:03.601342 7087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:03.601527 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:03.602082 7087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:03.603087 7087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:03.603168 7087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:03.603261 7087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:03.603313 7087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:03.612238 7087 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:33803
I20260924 16:12:03.612258 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:33803 every 8 connection(s)
I20260924 16:12:03.613200 7087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:03.613476 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 7087
I20260924 16:12:03.613608 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:03.613930 2037 external_mini_cluster.cc:1428] Setting key 265c2679eb2d4a74b96e3f1f29766423
I20260924 16:12:03.620185 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.615340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:49513 (local address 127.1.253.126:39317)
0924 16:12:03.615501 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:03.615506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:03.617010 (+ 1504us) server_negotiation.cc:407] Connection header received
0924 16:12:03.618411 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:03.618415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:03.618467 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:03.618549 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:03.619667 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:03.619672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:03.619674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:03.619688 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:03.619710 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:03.619719 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:03.619829 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:03.619980 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 16:12:03.620028 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 16:12:03.621186 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.615935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:39317 (local address 127.1.253.67:49513)
0924 16:12:03.616857 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:03.616906 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:03.618152 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:03.618732 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:03.618740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:03.618843 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:03.619393 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:03.619395 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:03.619438 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:03.619516 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:03.619813 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:03.619821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:03.619827 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:12:03.620026 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 16:12:03.620299 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":146,"threads_started":1}
I20260924 16:12:03.622331 7216 heartbeater.cc:347] Connected to a master server at 127.1.253.126:39317
I20260924 16:12:03.622605 7216 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:03.623063 7216 heartbeater.cc:511] Master 127.1.253.126:39317 requested a full tablet report, sending...
I20260924 16:12:03.624080 6764 ts_manager.cc:194] Registered new tserver with Master: 6c4bafe8bbd54184ae6400104f88ba00 (127.1.253.67:33803)
I20260924 16:12:03.628259 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:12:03.631888 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.630361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.1.253.126:39317)
0924 16:12:03.630538 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:03.630542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:03.630698 (+ 156us) server_negotiation.cc:407] Connection header received
0924 16:12:03.630787 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:03.630790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:03.630823 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:03.630890 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:03.631629 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:03.631631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:03.631633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:03.631641 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:03.631655 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:03.631662 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:03.631723 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:03.631765 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 16:12:03.631795 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 16:12:03.634626 6764 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:03.637732 6764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:03.650441 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.646802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:34937 (local address 127.0.0.1:56922)
0924 16:12:03.647290 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:03.647340 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:03.647446 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:03.648830 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:03.648836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:03.648920 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:03.649644 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:03.649645 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:03.649653 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:03.649693 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:03.650053 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:03.650062 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:03.650069 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:12:03.650203 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 16:12:03.650295 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":88,"threads_started":1}
I20260924 16:12:03.650507 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.646948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.1.253.66:34937)
0924 16:12:03.647592 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:03.647606 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:03.647632 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:12:03.647694 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:03.647702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:03.647933 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:03.648083 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:03.649868 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:03.649874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:03.649889 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:03.649899 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:03.649936 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:03.649963 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:03.650081 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:03.650233 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 16:12:03.650327 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":108,"threads_started":1}
I20260924 16:12:03.650882 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.647007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:33803 (local address 127.0.0.1:57032)
0924 16:12:03.647976 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:03.647992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:03.648058 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:03.648830 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:03.648836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:03.648920 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:03.649579 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:03.649580 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:03.649619 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:03.649694 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:03.650291 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:03.650294 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:03.650298 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:12:03.650369 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 16:12:03.650410 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":118,"threads_started":1}
I20260924 16:12:03.650942 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.647615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:42209 (local address 127.0.0.1:52056)
0924 16:12:03.648348 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:03.648373 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:03.648444 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:03.649222 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:03.649226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:03.649237 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:03.649826 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:03.649828 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:03.649839 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:03.649869 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:03.650701 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:03.650704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:03.650708 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:12:03.650781 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 16:12:03.650815 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":101,"threads_started":1}
I20260924 16:12:03.651427 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.647648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.1.253.67:33803)
0924 16:12:03.648235 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:03.648248 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:03.648274 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:12:03.648336 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:03.648341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:03.648525 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:03.648803 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:03.650019 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:03.650027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:03.650043 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:03.650058 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:03.650104 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:03.650138 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:03.650476 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:03.650538 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 16:12:03.650609 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":138,"threads_started":1}
I20260924 16:12:03.651499 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.647819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.1.253.65:42209)
0924 16:12:03.648694 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:03.648709 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:03.648733 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:12:03.648789 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:03.648794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:03.648959 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:03.649088 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:03.650435 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:03.650440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:03.650451 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:03.650475 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:03.650520 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:03.650547 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:03.650655 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:03.651238 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 16:12:03.651314 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":106,"threads_started":1}
I20260924 16:12:03.653923 7017 tablet_service.cc:1467] Processing CreateTablet for tablet d3d28233f4234de987160a9a61469c2d (DEFAULT_TABLE table=test-table [id=5b38c6fd22a944189e5d676ac7ba13c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:03.653921 7150 tablet_service.cc:1467] Processing CreateTablet for tablet d3d28233f4234de987160a9a61469c2d (DEFAULT_TABLE table=test-table [id=5b38c6fd22a944189e5d676ac7ba13c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:03.653863 6886 tablet_service.cc:1467] Processing CreateTablet for tablet d3d28233f4234de987160a9a61469c2d (DEFAULT_TABLE table=test-table [id=5b38c6fd22a944189e5d676ac7ba13c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:03.655030 6886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d28233f4234de987160a9a61469c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:03.655032 7150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d28233f4234de987160a9a61469c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:03.655032 7017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d28233f4234de987160a9a61469c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:03.661720 7233 tablet_bootstrap.cc:492] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d: Bootstrap starting.
I20260924 16:12:03.663038 7235 tablet_bootstrap.cc:492] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00: Bootstrap starting.
I20260924 16:12:03.663444 7234 tablet_bootstrap.cc:492] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c: Bootstrap starting.
I20260924 16:12:03.664371 7233 tablet_bootstrap.cc:654] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:03.665126 7233 log.cc:824] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:03.665261 7234 tablet_bootstrap.cc:654] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:03.666102 7234 log.cc:824] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:03.667418 7233 tablet_bootstrap.cc:492] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d: No bootstrap required, opened a new log
I20260924 16:12:03.667456 7235 tablet_bootstrap.cc:654] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:03.667652 7233 ts_tablet_manager.cc:1397] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:12:03.667992 7234 tablet_bootstrap.cc:492] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c: No bootstrap required, opened a new log
I20260924 16:12:03.668190 7234 ts_tablet_manager.cc:1397] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 16:12:03.668476 7235 log.cc:824] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:03.670660 7235 tablet_bootstrap.cc:492] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00: No bootstrap required, opened a new log
I20260924 16:12:03.670850 7235 ts_tablet_manager.cc:1397] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 16:12:03.671885 7233 raft_consensus.cc:348] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8254909a54fee8abf0a4652bef11c" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 34937 } } peers { permanent_uuid: "6c4bafe8bbd54184ae6400104f88ba00" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33803 } } peers { permanent_uuid: "8b79355e525e42fab3270608b3ebe88d" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42209 } }
I20260924 16:12:03.672145 7233 raft_consensus.cc:374] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:03.672206 7233 raft_consensus.cc:729] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b79355e525e42fab3270608b3ebe88d, State: Initialized, Role: FOLLOWER
I20260924 16:12:03.672720 7233 consensus_queue.cc:260] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8254909a54fee8abf0a4652bef11c" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 34937 } } peers { permanent_uuid: "6c4bafe8bbd54184ae6400104f88ba00" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33803 } } peers { permanent_uuid: "8b79355e525e42fab3270608b3ebe88d" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42209 } }
I20260924 16:12:03.672824 7234 raft_consensus.cc:348] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8254909a54fee8abf0a4652bef11c" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 34937 } } peers { permanent_uuid: "6c4bafe8bbd54184ae6400104f88ba00" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33803 } } peers { permanent_uuid: "8b79355e525e42fab3270608b3ebe88d" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42209 } }
I20260924 16:12:03.673106 7234 raft_consensus.cc:374] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:03.673157 7234 raft_consensus.cc:729] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a8254909a54fee8abf0a4652bef11c, State: Initialized, Role: FOLLOWER
I20260924 16:12:03.673694 6952 heartbeater.cc:503] Master 127.1.253.126:39317 was elected leader, sending a full tablet report...
I20260924 16:12:03.673630 7234 consensus_queue.cc:260] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8254909a54fee8abf0a4652bef11c" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 34937 } } peers { permanent_uuid: "6c4bafe8bbd54184ae6400104f88ba00" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33803 } } peers { permanent_uuid: "8b79355e525e42fab3270608b3ebe88d" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42209 } }
I20260924 16:12:03.673830 7233 ts_tablet_manager.cc:1428] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 16:12:03.674396 7084 heartbeater.cc:503] Master 127.1.253.126:39317 was elected leader, sending a full tablet report...
I20260924 16:12:03.674659 7234 ts_tablet_manager.cc:1428] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 16:12:03.676306 7235 raft_consensus.cc:348] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8254909a54fee8abf0a4652bef11c" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 34937 } } peers { permanent_uuid: "6c4bafe8bbd54184ae6400104f88ba00" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33803 } } peers { permanent_uuid: "8b79355e525e42fab3270608b3ebe88d" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42209 } }
I20260924 16:12:03.676612 7235 raft_consensus.cc:374] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:03.676681 7235 raft_consensus.cc:729] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4bafe8bbd54184ae6400104f88ba00, State: Initialized, Role: FOLLOWER
I20260924 16:12:03.677177 7235 consensus_queue.cc:260] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8254909a54fee8abf0a4652bef11c" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 34937 } } peers { permanent_uuid: "6c4bafe8bbd54184ae6400104f88ba00" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33803 } } peers { permanent_uuid: "8b79355e525e42fab3270608b3ebe88d" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42209 } }
I20260924 16:12:03.678124 7235 ts_tablet_manager.cc:1428] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 16:12:03.678426 7216 heartbeater.cc:503] Master 127.1.253.126:39317 was elected leader, sending a full tablet report...
I20260924 16:12:03.743274 7241 raft_consensus.cc:482] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:03.743544 7241 raft_consensus.cc:504] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8254909a54fee8abf0a4652bef11c" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 34937 } } peers { permanent_uuid: "6c4bafe8bbd54184ae6400104f88ba00" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33803 } } peers { permanent_uuid: "8b79355e525e42fab3270608b3ebe88d" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42209 } }
W20260924 16:12:03.744206 6953 tablet.cc:2369] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:03.745002 7241 leader_election.cc:288] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44a8254909a54fee8abf0a4652bef11c (127.1.253.66:34937), 8b79355e525e42fab3270608b3ebe88d (127.1.253.65:42209)
I20260924 16:12:03.747000 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.745389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:38539 (local address 127.1.253.65:42209)
0924 16:12:03.745569 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:03.745577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:03.745595 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:12:03.745645 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:03.745648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:03.745698 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:03.745789 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:03.746593 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:03.746597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:03.746599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:03.746612 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:03.746638 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:03.746651 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:03.746740 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:03.746826 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 16:12:03.746862 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:12:03.747020 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.745266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:42209 (local address 127.1.253.67:38539)
0924 16:12:03.745424 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:03.745441 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:03.745529 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:03.745833 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:03.745837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:03.745854 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:03.746414 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:03.746418 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:03.746432 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:03.746465 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:03.746737 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:03.746740 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:03.746744 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:12:03.746823 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 16:12:03.746868 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 16:12:03.747726 6906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d28233f4234de987160a9a61469c2d" candidate_uuid: "6c4bafe8bbd54184ae6400104f88ba00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b79355e525e42fab3270608b3ebe88d" is_pre_election: true
I20260924 16:12:03.747929 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.745391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:58103 (local address 127.1.253.66:34937)
0924 16:12:03.745544 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:03.745549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:03.745952 (+ 403us) server_negotiation.cc:407] Connection header received
0924 16:12:03.746565 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:03.746569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:03.746611 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:03.746703 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:03.747548 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:03.747552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:03.747554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:03.747567 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:03.747588 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:03.747598 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:03.747690 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:03.747770 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 16:12:03.747811 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:12:03.747934 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.745266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:34937 (local address 127.1.253.67:58103)
0924 16:12:03.745833 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:03.745847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:03.745934 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:03.746744 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:03.746747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:03.746764 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:03.747407 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:03.747409 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:03.747417 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:03.747439 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:03.747683 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:03.747685 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:03.747687 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 16:12:03.747761 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 16:12:03.747791 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":187,"thread_start_us":120,"threads_started":1}
I20260924 16:12:03.748076 6906 raft_consensus.cc:2454] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c4bafe8bbd54184ae6400104f88ba00 in term 0.
I20260924 16:12:03.748562 7103 leader_election.cc:302] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c4bafe8bbd54184ae6400104f88ba00, 8b79355e525e42fab3270608b3ebe88d; no voters:
I20260924 16:12:03.748855 7241 raft_consensus.cc:2781] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:03.748973 7241 raft_consensus.cc:482] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:03.748939 7037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d28233f4234de987160a9a61469c2d" candidate_uuid: "6c4bafe8bbd54184ae6400104f88ba00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a8254909a54fee8abf0a4652bef11c" is_pre_election: true
I20260924 16:12:03.749042 7241 raft_consensus.cc:3037] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:03.749250 7037 raft_consensus.cc:2454] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c4bafe8bbd54184ae6400104f88ba00 in term 0.
I20260924 16:12:03.776608 7241 raft_consensus.cc:504] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8254909a54fee8abf0a4652bef11c" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 34937 } } peers { permanent_uuid: "6c4bafe8bbd54184ae6400104f88ba00" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33803 } } peers { permanent_uuid: "8b79355e525e42fab3270608b3ebe88d" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42209 } }
I20260924 16:12:03.777200 7241 leader_election.cc:288] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [CANDIDATE]: Term 1 election: Requested vote from peers 44a8254909a54fee8abf0a4652bef11c (127.1.253.66:34937), 8b79355e525e42fab3270608b3ebe88d (127.1.253.65:42209)
I20260924 16:12:03.777770 6906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d28233f4234de987160a9a61469c2d" candidate_uuid: "6c4bafe8bbd54184ae6400104f88ba00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b79355e525e42fab3270608b3ebe88d"
I20260924 16:12:03.777774 7037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d28233f4234de987160a9a61469c2d" candidate_uuid: "6c4bafe8bbd54184ae6400104f88ba00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a8254909a54fee8abf0a4652bef11c"
I20260924 16:12:03.777956 6906 raft_consensus.cc:3037] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:03.777946 7037 raft_consensus.cc:3037] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:03.779227 6906 raft_consensus.cc:2454] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c4bafe8bbd54184ae6400104f88ba00 in term 1.
I20260924 16:12:03.779687 7103 leader_election.cc:302] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c4bafe8bbd54184ae6400104f88ba00, 8b79355e525e42fab3270608b3ebe88d; no voters:
I20260924 16:12:03.779726 7037 raft_consensus.cc:2454] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c4bafe8bbd54184ae6400104f88ba00 in term 1.
I20260924 16:12:03.779891 7241 raft_consensus.cc:2781] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:03.780170 7241 raft_consensus.cc:686] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 1 LEADER]: Becoming Leader. State: Replica: 6c4bafe8bbd54184ae6400104f88ba00, State: Running, Role: LEADER
I20260924 16:12:03.780622 7241 consensus_queue.cc:237] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8254909a54fee8abf0a4652bef11c" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 34937 } } peers { permanent_uuid: "6c4bafe8bbd54184ae6400104f88ba00" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33803 } } peers { permanent_uuid: "8b79355e525e42fab3270608b3ebe88d" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42209 } }
I20260924 16:12:03.784104 6764 catalog_manager.cc:5495] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c4bafe8bbd54184ae6400104f88ba00 (127.1.253.67). New cstate: current_term: 1 leader_uuid: "6c4bafe8bbd54184ae6400104f88ba00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8254909a54fee8abf0a4652bef11c" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 34937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c4bafe8bbd54184ae6400104f88ba00" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b79355e525e42fab3270608b3ebe88d" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42209 } health_report { overall_health: UNKNOWN } } }
I20260924 16:12:03.804148 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.802608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.1.253.67:33803)
0924 16:12:03.802754 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:03.802761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:03.802777 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:12:03.802869 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:03.802871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:03.802933 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:03.803025 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:03.803731 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:03.803735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:03.803738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:03.803749 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:03.803768 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:03.803780 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:03.803879 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:03.803949 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 16:12:03.803992 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:12:03.810853 7037 raft_consensus.cc:1261] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c [term 1 FOLLOWER]: Refusing update from remote peer 6c4bafe8bbd54184ae6400104f88ba00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:03.810847 6906 raft_consensus.cc:1261] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d [term 1 FOLLOWER]: Refusing update from remote peer 6c4bafe8bbd54184ae6400104f88ba00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:03.811625 7241 consensus_queue.cc:1035] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b79355e525e42fab3270608b3ebe88d" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:03.811837 7243 consensus_queue.cc:1035] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a8254909a54fee8abf0a4652bef11c" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 34937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:03.820406 7248 mvcc.cc:205] Tried to move back new op lower bound from 7332930862321471488 to 7332930862208356352. Current Snapshot: MvccSnapshot[applied={T|T < 7332930862321471488}]
I20260924 16:12:03.820418 7247 mvcc.cc:205] Tried to move back new op lower bound from 7332930862321471488 to 7332930862208356352. Current Snapshot: MvccSnapshot[applied={T|T < 7332930862321471488}]
I20260924 16:12:03.821696 7246 mvcc.cc:205] Tried to move back new op lower bound from 7332930862321471488 to 7332930862208356352. Current Snapshot: MvccSnapshot[applied={T|T < 7332930862321471488}]
I20260924 16:12:03.829835 6764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:03.830030 6764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:03.832777 6764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c757dacd100445fbba0999e87674c5a5: Sending DeleteTablet for 3 replicas of tablet d3d28233f4234de987160a9a61469c2d
I20260924 16:12:03.833637 7017 tablet_service.cc:1514] Processing DeleteTablet for tablet d3d28233f4234de987160a9a61469c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:03 UTC) from {username='slave'} at 127.0.0.1:56922
I20260924 16:12:03.833673 7150 tablet_service.cc:1514] Processing DeleteTablet for tablet d3d28233f4234de987160a9a61469c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:03 UTC) from {username='slave'} at 127.0.0.1:57032
I20260924 16:12:03.834227 7257 tablet_replica.cc:330] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c: stopping tablet replica
I20260924 16:12:03.834312 6886 tablet_service.cc:1514] Processing DeleteTablet for tablet d3d28233f4234de987160a9a61469c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:03 UTC) from {username='slave'} at 127.0.0.1:52056
I20260924 16:12:03.834574 7256 tablet_replica.cc:330] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00: stopping tablet replica
I20260924 16:12:03.834795 7258 tablet_replica.cc:330] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d: stopping tablet replica
I20260924 16:12:03.834947 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 6824
I20260924 16:12:03.835119 7258 raft_consensus.cc:2229] T d3d28233f4234de987160a9a61469c2d P 8b79355e525e42fab3270608b3ebe88d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:03.836253 7256 raft_consensus.cc:2229] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:03.836730 7256 raft_consensus.cc:2258] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:03.837040 7257 raft_consensus.cc:2229] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:03.837361 7257 raft_consensus.cc:2258] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:03.838528 7256 ts_tablet_manager.cc:1905] T d3d28233f4234de987160a9a61469c2d P 6c4bafe8bbd54184ae6400104f88ba00: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 16:12:03.844046 6748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:42209 (error 108)
I20260924 16:12:03.844130 7257 ts_tablet_manager.cc:1905] T d3d28233f4234de987160a9a61469c2d P 44a8254909a54fee8abf0a4652bef11c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:03.845147 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.844814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:42209 (local address 127.0.0.1:52066)
0924 16:12:03.844957 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:03.845038 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:42209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 16:12:03.845212 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 6955
W20260924 16:12:03.847137 6748 catalog_manager.cc:4584] TS 8b79355e525e42fab3270608b3ebe88d (127.1.253.65:42209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3d28233f4234de987160a9a61469c2d: Network error: Client connection negotiation failed: client connection to 127.1.253.65:42209: connect: Connection refused (error 111)
I20260924 16:12:03.879014 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 7087
I20260924 16:12:03.879313 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:03.878988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:34937 (local address 127.0.0.1:56926)
0924 16:12:03.879135 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:03.879200 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.66:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 16:12:03.890542 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 6732
2026-09-24T16:12:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1097 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 16:12:03.923601 2037 mini_oidc.cc:152] Starting JWKS server
I20260924 16:12:03.925293 2037 webserver.cc:460] Webserver started at https://127.0.0.1:33943/ using document root <none> and password file <none>
I20260924 16:12:03.925500 2037 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 16:12:03.925781 2037 webserver.cc:460] Webserver started at http://127.0.0.1:44705/ using document root <none> and password file <none>
2026-09-24T16:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:03Z Disabled control of system clock
I20260924 16:12:03.935431 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:37289
--webserver_interface=127.1.253.126
--webserver_port=0
--builtin_ntp_servers=127.1.253.84:35055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:37289
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33943/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/testchainca.cert with env {}
W20260924 16:12:04.049646 7265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:04.049891 7265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:04.049932 7265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:04.053972 7265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:04.054044 7265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:04.054059 7265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:04.054075 7265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:04.054090 7265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:12:04.054144 7265 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:12:04.054163 7265 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33943/jwks.json
I20260924 16:12:04.059381 7265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:37289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:37289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33943/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:04.060410 7265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:04.061411 7265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:04.069811 7270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:04.069830 7273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:04.069849 7265 server_base.cc:1034] running on GCE node
W20260924 16:12:04.069801 7271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:04.070621 7265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:04.071652 7265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:04.072837 7265 hybrid_clock.cc:648] HybridClock initialized: now 1790266324072804 us; error 44 us; skew 500 ppm
I20260924 16:12:04.074810 7265 webserver.cc:460] Webserver started at http://127.1.253.126:35469/ using document root <none> and password file <none>
I20260924 16:12:04.075377 7265 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:04.075471 7265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:04.075697 7265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:04.077623 7265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "54ae1da57f8e4084b34d24311d99f34c"
format_stamp: "Formatted at 2026-09-24 16:12:04 on dist-test-slave-6wvm"
server_key: "595fb1e98c6e9cbd200f938486e0b840"
server_key_iv: "fda711d666104fec65ab09b94f9e3eb9"
server_key_version: "encryptionkey@0"
I20260924 16:12:04.078191 7265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "54ae1da57f8e4084b34d24311d99f34c"
format_stamp: "Formatted at 2026-09-24 16:12:04 on dist-test-slave-6wvm"
server_key: "595fb1e98c6e9cbd200f938486e0b840"
server_key_iv: "fda711d666104fec65ab09b94f9e3eb9"
server_key_version: "encryptionkey@0"
I20260924 16:12:04.082162 7265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:12:04.084878 7279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:04.086619 7265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:04.086753 7265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "54ae1da57f8e4084b34d24311d99f34c"
format_stamp: "Formatted at 2026-09-24 16:12:04 on dist-test-slave-6wvm"
server_key: "595fb1e98c6e9cbd200f938486e0b840"
server_key_iv: "fda711d666104fec65ab09b94f9e3eb9"
server_key_version: "encryptionkey@0"
I20260924 16:12:04.086880 7265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:04.109218 7265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:04.109997 7265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:04.110215 7265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:04.117245 7265 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:37289
I20260924 16:12:04.117250 7331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:37289 every 8 connection(s)
I20260924 16:12:04.118219 7265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:04.120842 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 7265
I20260924 16:12:04.120996 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:04.121277 2037 external_mini_cluster.cc:1428] Setting key 73759bc3a644b6970a25b9aeacca926a
I20260924 16:12:04.122434 7332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:04.128495 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c: Bootstrap starting.
I20260924 16:12:04.130496 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:04.122670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.1.253.126:37289)
0924 16:12:04.123334 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:04.123349 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:04.123384 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:12:04.124028 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:04.124045 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:04.124420 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:04.124750 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:04.125169 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.126175 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.127125 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.127592 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.127764 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:04.128734 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:04.128739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:04.128750 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:04.129126 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:04.129180 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:04.129199 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:04.129310 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:04.129461 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 16:12:04.129723 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
I20260924 16:12:04.131057 7332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:04.131851 7332 log.cc:824] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:04.133450 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:12:04.133921 7332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c: No bootstrap required, opened a new log
I20260924 16:12:04.137578 7332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ae1da57f8e4084b34d24311d99f34c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 37289 } }
I20260924 16:12:04.137825 7332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:04.137908 7332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ae1da57f8e4084b34d24311d99f34c, State: Initialized, Role: FOLLOWER
I20260924 16:12:04.138413 7332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ae1da57f8e4084b34d24311d99f34c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 37289 } }
I20260924 16:12:04.138581 7332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:04.138659 7332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:04.138785 7332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:04.176909 7332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ae1da57f8e4084b34d24311d99f34c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 37289 } }
I20260924 16:12:04.177444 7332 leader_election.cc:302] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54ae1da57f8e4084b34d24311d99f34c; no voters:
I20260924 16:12:04.177991 7332 leader_election.cc:288] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:04.178151 7337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:04.178421 7337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [term 1 LEADER]: Becoming Leader. State: Replica: 54ae1da57f8e4084b34d24311d99f34c, State: Running, Role: LEADER
I20260924 16:12:04.178828 7337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ae1da57f8e4084b34d24311d99f34c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 37289 } }
I20260924 16:12:04.179725 7332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:04.180334 7339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54ae1da57f8e4084b34d24311d99f34c. Latest consensus state: current_term: 1 leader_uuid: "54ae1da57f8e4084b34d24311d99f34c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ae1da57f8e4084b34d24311d99f34c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 37289 } } }
I20260924 16:12:04.180480 7339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:04.180702 7338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54ae1da57f8e4084b34d24311d99f34c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ae1da57f8e4084b34d24311d99f34c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 37289 } } }
I20260924 16:12:04.180838 7338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:04.180996 7341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:04.184516 7341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:04.190454 7341 catalog_manager.cc:1347] Generated new cluster ID: d23507f077f44fec97d2c13228a3acae
I20260924 16:12:04.190531 7341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:04.209689 7341 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:04.210731 7341 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:04.214699 7341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54ae1da57f8e4084b34d24311d99f34c: Generated new TSK 0
I20260924 16:12:04.215387 7341 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 16:12:04.254499 7362 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 16:12:04.254652 7362 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 16:12:04.255056 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:04.235493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.1.253.126:37289)
0924 16:12:04.235645 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:04.235650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:04.235829 (+ 179us) server_negotiation.cc:407] Connection header received
0924 16:12:04.236082 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:04.236086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:04.236148 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:04.236266 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:12:04.236550 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.237369 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.238275 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.238643 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.238748 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:04.239085 (+ 337us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:12:04.254616 (+ 15531us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33943/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 16:12:04.254905 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33943/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 16:12:04.255141 7335 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33943/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 16:12:04.256733 7361 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:04.235361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:37289 (local address 127.0.0.1:32802)
0924 16:12:04.235748 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:04.235761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:04.235910 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:04.236288 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:04.236291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:04.236299 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:12:04.236442 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:04.236448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.237474 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.237477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:04.238131 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:04.238148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.238772 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.238774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:04.238844 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:04.238890 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:12:04.254908 (+ 16018us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33943/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 16:12:04.255148 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.253.126:37289: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33943/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":116,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.253.126:37289: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33943/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 16:12:04.257519 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 7265
2026-09-24T16:12:04Z chronyd exiting
I20260924 16:12:04.296928 2037 test_util.cc:175] -----------------------------------------------
I20260924 16:12:04.297029 2037 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266297063660-2037-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (375 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 16:12:04.298041 2037 mini_oidc.cc:152] Starting JWKS server
I20260924 16:12:04.298969 2037 webserver.cc:460] Webserver started at https://127.0.0.1:40699/ using document root <none> and password file <none>
I20260924 16:12:04.299052 2037 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 16:12:04.299263 2037 webserver.cc:460] Webserver started at http://127.0.0.1:41045/ using document root <none> and password file <none>
2026-09-24T16:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:04Z Disabled control of system clock
I20260924 16:12:04.309056 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:42567
--webserver_interface=127.1.253.126
--webserver_port=0
--builtin_ntp_servers=127.1.253.84:35423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:42567
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40699/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/testchainca.cert with env {}
W20260924 16:12:04.423602 7368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:04.423867 7368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:04.423972 7368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:04.427937 7368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:04.427992 7368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:04.428009 7368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:04.428025 7368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:04.428041 7368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:12:04.428056 7368 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:12:04.428073 7368 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40699/jwks.json
I20260924 16:12:04.433388 7368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:42567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:42567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40699/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:04.434577 7368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:04.435611 7368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:04.444367 7373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:04.444375 7374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:04.444690 7368 server_base.cc:1034] running on GCE node
W20260924 16:12:04.444389 7376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:04.445504 7368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:04.446712 7368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:04.447876 7368 hybrid_clock.cc:648] HybridClock initialized: now 1790266324447849 us; error 40 us; skew 500 ppm
I20260924 16:12:04.449759 7368 webserver.cc:460] Webserver started at http://127.1.253.126:38803/ using document root <none> and password file <none>
I20260924 16:12:04.450418 7368 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:04.450485 7368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:04.450742 7368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:04.452854 7368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "940a89343d1844debeb934fcd0696d5d"
format_stamp: "Formatted at 2026-09-24 16:12:04 on dist-test-slave-6wvm"
server_key: "3cb2adf0bae397877c6efc0759aea248"
server_key_iv: "4c5d8a8898163ba239199d0fe9c14519"
server_key_version: "encryptionkey@0"
I20260924 16:12:04.453437 7368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "940a89343d1844debeb934fcd0696d5d"
format_stamp: "Formatted at 2026-09-24 16:12:04 on dist-test-slave-6wvm"
server_key: "3cb2adf0bae397877c6efc0759aea248"
server_key_iv: "4c5d8a8898163ba239199d0fe9c14519"
server_key_version: "encryptionkey@0"
I20260924 16:12:04.457226 7368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:12:04.459760 7382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:04.461061 7368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:12:04.461213 7368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "940a89343d1844debeb934fcd0696d5d"
format_stamp: "Formatted at 2026-09-24 16:12:04 on dist-test-slave-6wvm"
server_key: "3cb2adf0bae397877c6efc0759aea248"
server_key_iv: "4c5d8a8898163ba239199d0fe9c14519"
server_key_version: "encryptionkey@0"
I20260924 16:12:04.461329 7368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:04.475718 7368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:04.476397 7368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:04.476588 7368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:04.483879 7368 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:42567
I20260924 16:12:04.483886 7434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:42567 every 8 connection(s)
I20260924 16:12:04.484859 7368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:04.485141 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 7368
I20260924 16:12:04.485265 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:04.485669 2037 external_mini_cluster.cc:1428] Setting key 169887da90c9bdad5644d62d73848862
I20260924 16:12:04.488718 7435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:04.494697 7435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d: Bootstrap starting.
I20260924 16:12:04.495149 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:04.487159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.1.253.126:42567)
0924 16:12:04.487850 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:04.487865 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:04.487901 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:12:04.488540 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:04.488556 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:04.489016 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:04.489364 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:04.489798 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.490845 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.491747 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.492229 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.492402 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:04.493321 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:04.493326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:04.493338 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:04.493727 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:04.493781 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:04.493800 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:04.493912 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:04.494071 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 16:12:04.494385 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":155,"threads_started":1}
I20260924 16:12:04.497263 7435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:04.498147 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:12:04.498209 7435 log.cc:824] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:04.500427 7435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d: No bootstrap required, opened a new log
I20260924 16:12:04.503859 7435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940a89343d1844debeb934fcd0696d5d" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42567 } }
I20260924 16:12:04.504063 7435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:04.504154 7435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 940a89343d1844debeb934fcd0696d5d, State: Initialized, Role: FOLLOWER
I20260924 16:12:04.504676 7435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940a89343d1844debeb934fcd0696d5d" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42567 } }
I20260924 16:12:04.504878 7435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:04.504952 7435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:04.505095 7435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:04.575515 7435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940a89343d1844debeb934fcd0696d5d" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42567 } }
I20260924 16:12:04.576184 7435 leader_election.cc:302] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 940a89343d1844debeb934fcd0696d5d; no voters:
I20260924 16:12:04.576759 7435 leader_election.cc:288] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:04.576855 7440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:04.577105 7440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [term 1 LEADER]: Becoming Leader. State: Replica: 940a89343d1844debeb934fcd0696d5d, State: Running, Role: LEADER
I20260924 16:12:04.577477 7440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940a89343d1844debeb934fcd0696d5d" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42567 } }
I20260924 16:12:04.578446 7435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:04.579457 7441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "940a89343d1844debeb934fcd0696d5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940a89343d1844debeb934fcd0696d5d" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42567 } } }
I20260924 16:12:04.579603 7441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:04.579514 7442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 940a89343d1844debeb934fcd0696d5d. Latest consensus state: current_term: 1 leader_uuid: "940a89343d1844debeb934fcd0696d5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940a89343d1844debeb934fcd0696d5d" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42567 } } }
I20260924 16:12:04.579660 7442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d [sys.catalog]: This master's current role is: LEADER
W20260924 16:12:04.583779 7455 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:12:04.583850 7455 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:12:04.583931 7449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:04.584656 7449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:04.589942 7449 catalog_manager.cc:1347] Generated new cluster ID: 6ef8d105b6804afaa2232a3b4849c5b1
I20260924 16:12:04.590003 7449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:04.601076 7449 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:04.602414 7449 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:04.617470 7449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 940a89343d1844debeb934fcd0696d5d: Generated new TSK 0
I20260924 16:12:04.618283 7449 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 16:12:04.645314 7465 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 16:12:04.645416 7465 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 16:12:04.646137 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:04.629783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49048 (local address 127.1.253.126:42567)
0924 16:12:04.629934 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:04.629940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:04.630147 (+ 207us) server_negotiation.cc:407] Connection header received
0924 16:12:04.630367 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:04.630370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:04.630422 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:04.630517 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:12:04.630830 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.631362 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.632598 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.632799 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.632873 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:04.633227 (+ 354us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:12:04.645765 (+ 12538us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 16:12:04.646013 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49048: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 16:12:04.646232 7438 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49048: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 16:12:04.646468 7464 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:04.629695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42567 (local address 127.0.0.1:49048)
0924 16:12:04.630019 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:04.630035 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:04.630162 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:04.630545 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:04.630550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:04.630560 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:12:04.630712 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:04.630720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.631473 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.631477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:04.632475 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:04.632486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.632907 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.632911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:04.633025 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:04.633055 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:12:04.646000 (+ 12945us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 16:12:04.646262 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.253.126:42567: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":111,"threads_started":1}
I20260924 16:12:04.647279 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 7368
2026-09-24T16:12:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (389 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 16:12:04.687974 2037 mini_oidc.cc:152] Starting JWKS server
I20260924 16:12:04.688885 2037 webserver.cc:460] Webserver started at https://127.0.0.1:35553/ using document root <none> and password file <none>
I20260924 16:12:04.689025 2037 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 16:12:04.689293 2037 webserver.cc:460] Webserver started at http://127.0.0.1:36025/ using document root <none> and password file <none>
2026-09-24T16:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:04Z Disabled control of system clock
I20260924 16:12:04.698802 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:35419
--webserver_interface=127.1.253.126
--webserver_port=0
--builtin_ntp_servers=127.1.253.84:37105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:35419
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35553/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 16:12:04.810400 7471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:04.810632 7471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:04.810685 7471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:04.814682 7471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:04.814731 7471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:04.814746 7471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:04.814795 7471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:04.814821 7471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:12:04.814857 7471 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:12:04.814893 7471 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35553/jwks.json
I20260924 16:12:04.819465 7471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:37105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:35419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35553/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:04.820490 7471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:04.821825 7471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:04.830312 7476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:04.830430 7471 server_base.cc:1034] running on GCE node
W20260924 16:12:04.830305 7477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:04.830312 7479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:04.831171 7471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:04.832314 7471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:04.833516 7471 hybrid_clock.cc:648] HybridClock initialized: now 1790266324833508 us; error 31 us; skew 500 ppm
I20260924 16:12:04.835479 7471 webserver.cc:460] Webserver started at http://127.1.253.126:35261/ using document root <none> and password file <none>
I20260924 16:12:04.836099 7471 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:04.836195 7471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:04.836437 7471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:04.838610 7471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "f2c2d3d7ef244180b980dd84ad2706be"
format_stamp: "Formatted at 2026-09-24 16:12:04 on dist-test-slave-6wvm"
server_key: "8783311b686086de5649066088f16b5d"
server_key_iv: "3c77bbda4c826d327ed888a4099fb8d9"
server_key_version: "encryptionkey@0"
I20260924 16:12:04.839200 7471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "f2c2d3d7ef244180b980dd84ad2706be"
format_stamp: "Formatted at 2026-09-24 16:12:04 on dist-test-slave-6wvm"
server_key: "8783311b686086de5649066088f16b5d"
server_key_iv: "3c77bbda4c826d327ed888a4099fb8d9"
server_key_version: "encryptionkey@0"
I20260924 16:12:04.843030 7471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:12:04.845418 7485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:04.846698 7471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:04.846846 7471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "f2c2d3d7ef244180b980dd84ad2706be"
format_stamp: "Formatted at 2026-09-24 16:12:04 on dist-test-slave-6wvm"
server_key: "8783311b686086de5649066088f16b5d"
server_key_iv: "3c77bbda4c826d327ed888a4099fb8d9"
server_key_version: "encryptionkey@0"
I20260924 16:12:04.846959 7471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:04.870227 7471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:04.871134 7471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:04.871363 7471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:04.879725 7471 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:35419
I20260924 16:12:04.879731 7537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:35419 every 8 connection(s)
I20260924 16:12:04.880854 7471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:04.884027 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 7471
I20260924 16:12:04.884192 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:04.884522 2037 external_mini_cluster.cc:1428] Setting key ada91b31424aacf47c632c4aa2db4177
I20260924 16:12:04.885785 7538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:04.891999 7538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be: Bootstrap starting.
I20260924 16:12:04.894330 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:04.886097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.1.253.126:35419)
0924 16:12:04.886826 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:04.886842 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:04.886879 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:12:04.887539 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:04.887556 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:04.887959 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:04.888337 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:04.888813 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.889840 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.890829 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:04.891302 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:04.891490 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:04.892513 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:04.892518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:04.892532 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:04.892924 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:04.892984 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:04.893004 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:04.893122 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:04.893257 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 16:12:04.893535 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":158,"threads_started":1}
I20260924 16:12:04.894577 7538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:04.895323 7538 log.cc:824] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:04.897389 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:12:04.897657 7538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be: No bootstrap required, opened a new log
I20260924 16:12:04.901345 7538 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c2d3d7ef244180b980dd84ad2706be" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35419 } }
I20260924 16:12:04.901594 7538 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:04.901682 7538 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c2d3d7ef244180b980dd84ad2706be, State: Initialized, Role: FOLLOWER
I20260924 16:12:04.902197 7538 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c2d3d7ef244180b980dd84ad2706be" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35419 } }
I20260924 16:12:04.902365 7538 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:04.902437 7538 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:04.902562 7538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:04.979401 7538 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c2d3d7ef244180b980dd84ad2706be" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35419 } }
I20260924 16:12:04.979979 7538 leader_election.cc:302] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2c2d3d7ef244180b980dd84ad2706be; no voters:
I20260924 16:12:04.980427 7538 leader_election.cc:288] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:04.980659 7543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:04.980911 7543 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [term 1 LEADER]: Becoming Leader. State: Replica: f2c2d3d7ef244180b980dd84ad2706be, State: Running, Role: LEADER
I20260924 16:12:04.981288 7543 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c2d3d7ef244180b980dd84ad2706be" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35419 } }
I20260924 16:12:04.981913 7538 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:04.983199 7544 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2c2d3d7ef244180b980dd84ad2706be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c2d3d7ef244180b980dd84ad2706be" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35419 } } }
I20260924 16:12:04.983353 7544 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:04.983536 7545 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2c2d3d7ef244180b980dd84ad2706be. Latest consensus state: current_term: 1 leader_uuid: "f2c2d3d7ef244180b980dd84ad2706be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c2d3d7ef244180b980dd84ad2706be" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35419 } } }
I20260924 16:12:04.983706 7545 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:04.983767 7552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:04.987242 7552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:04.993156 7552 catalog_manager.cc:1347] Generated new cluster ID: eae0af598e3340aab782ca3ef2d459ab
I20260924 16:12:04.993227 7552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:05.002853 7552 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:05.004073 7552 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:05.009009 7552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2c2d3d7ef244180b980dd84ad2706be: Generated new TSK 0
I20260924 16:12:05.009879 7552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:12:05.027503 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:05.022137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45488 (local address 127.1.253.126:35419)
0924 16:12:05.022304 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:05.022309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:05.022535 (+ 226us) server_negotiation.cc:407] Connection header received
0924 16:12:05.022676 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:05.022679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:05.022740 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:05.022865 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:12:05.023078 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.023883 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.024777 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.025123 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.025227 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:05.025535 (+ 308us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:12:05.027109 (+ 1574us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35553/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 16:12:05.027348 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35553/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 16:12:05.027576 7541 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35553/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 16:12:05.029356 7567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:05.022001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:35419 (local address 127.0.0.1:45488)
0924 16:12:05.022446 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:05.022459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:05.022558 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:05.022868 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:05.022870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:05.022877 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:12:05.022972 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:05.022977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.024029 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.024031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:05.024659 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:05.024666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.025250 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.025253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:05.025319 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:05.025343 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:12:05.027289 (+ 1946us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35553/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 16:12:05.029178 (+ 1889us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.253.126:35419: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35553/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":1681,"thread_start_us":126,"threads_started":1}
I20260924 16:12:05.030251 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 7471
E20260924 16:12:05.039209 7568 webserver.cc:533] Webserver: sslize failed:
E20260924 16:12:05.039278 7568 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T16:12:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (382 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T16:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:05Z Disabled control of system clock
I20260924 16:12:05.080345 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:38547
--webserver_interface=127.1.253.126
--webserver_port=0
--builtin_ntp_servers=127.1.253.84:32851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:38547
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/testchainca.cert with env {}
W20260924 16:12:05.192703 7572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:05.192974 7572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:05.193025 7572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:05.197022 7572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:05.197077 7572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:05.197093 7572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:05.197109 7572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:05.197124 7572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:12:05.197140 7572 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:12:05.197161 7572 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 16:12:05.202406 7572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:32851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:38547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:38547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:05.203379 7572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:05.204345 7572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:05.212953 7578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:05.212955 7577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:05.213027 7580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:05.213104 7572 server_base.cc:1034] running on GCE node
I20260924 16:12:05.213871 7572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:05.215049 7572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:05.216272 7572 hybrid_clock.cc:648] HybridClock initialized: now 1790266325216243 us; error 49 us; skew 500 ppm
I20260924 16:12:05.218385 7572 webserver.cc:460] Webserver started at http://127.1.253.126:42355/ using document root <none> and password file <none>
I20260924 16:12:05.218999 7572 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:05.219094 7572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:05.219336 7572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:05.221498 7572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "2e4bb7bb0624453cbaffb55ee6675853"
format_stamp: "Formatted at 2026-09-24 16:12:05 on dist-test-slave-6wvm"
server_key: "ef704946ca26a0d8b97b9a97320d5163"
server_key_iv: "5796d77d99b669b415a75d913285e5b8"
server_key_version: "encryptionkey@0"
I20260924 16:12:05.222110 7572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "2e4bb7bb0624453cbaffb55ee6675853"
format_stamp: "Formatted at 2026-09-24 16:12:05 on dist-test-slave-6wvm"
server_key: "ef704946ca26a0d8b97b9a97320d5163"
server_key_iv: "5796d77d99b669b415a75d913285e5b8"
server_key_version: "encryptionkey@0"
I20260924 16:12:05.225973 7572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260924 16:12:05.228530 7586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:05.229839 7572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:05.229987 7572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "2e4bb7bb0624453cbaffb55ee6675853"
format_stamp: "Formatted at 2026-09-24 16:12:05 on dist-test-slave-6wvm"
server_key: "ef704946ca26a0d8b97b9a97320d5163"
server_key_iv: "5796d77d99b669b415a75d913285e5b8"
server_key_version: "encryptionkey@0"
I20260924 16:12:05.230110 7572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:05.245354 7572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:05.246078 7572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:05.246335 7572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:05.253190 7572 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:38547
I20260924 16:12:05.253201 7638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:38547 every 8 connection(s)
I20260924 16:12:05.254132 7572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:05.256048 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 7572
I20260924 16:12:05.256186 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:05.256523 2037 external_mini_cluster.cc:1428] Setting key c55a636ce00c8af29351b0bd18277b49
I20260924 16:12:05.257892 7639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:05.263585 7639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853: Bootstrap starting.
I20260924 16:12:05.265656 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:05.257918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.1.253.126:38547)
0924 16:12:05.258626 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:05.258646 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:05.258683 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:12:05.259334 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:05.259350 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:05.259742 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:05.260093 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:05.260535 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.261482 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.262426 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.262887 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.263083 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:05.263988 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:05.263994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:05.264007 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:05.264383 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:05.264442 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:05.264464 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:05.264574 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:05.264692 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 16:12:05.264958 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":143,"threads_started":1}
I20260924 16:12:05.265987 7639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:05.266867 7639 log.cc:824] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:05.268764 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:12:05.269164 7639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853: No bootstrap required, opened a new log
I20260924 16:12:05.273051 7639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4bb7bb0624453cbaffb55ee6675853" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38547 } }
I20260924 16:12:05.273273 7639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:05.273350 7639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4bb7bb0624453cbaffb55ee6675853, State: Initialized, Role: FOLLOWER
I20260924 16:12:05.273852 7639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4bb7bb0624453cbaffb55ee6675853" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38547 } }
I20260924 16:12:05.274019 7639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:05.274104 7639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:05.274214 7639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:05.275256 7639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4bb7bb0624453cbaffb55ee6675853" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38547 } }
I20260924 16:12:05.275622 7639 leader_election.cc:302] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e4bb7bb0624453cbaffb55ee6675853; no voters:
I20260924 16:12:05.275975 7639 leader_election.cc:288] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:05.276089 7644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:05.276343 7644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [term 1 LEADER]: Becoming Leader. State: Replica: 2e4bb7bb0624453cbaffb55ee6675853, State: Running, Role: LEADER
I20260924 16:12:05.276661 7644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4bb7bb0624453cbaffb55ee6675853" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38547 } }
I20260924 16:12:05.277264 7639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:05.278414 7646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e4bb7bb0624453cbaffb55ee6675853. Latest consensus state: current_term: 1 leader_uuid: "2e4bb7bb0624453cbaffb55ee6675853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4bb7bb0624453cbaffb55ee6675853" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38547 } } }
I20260924 16:12:05.278376 7645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e4bb7bb0624453cbaffb55ee6675853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4bb7bb0624453cbaffb55ee6675853" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38547 } } }
I20260924 16:12:05.278527 7646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:05.278548 7645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853 [sys.catalog]: This master's current role is: LEADER
W20260924 16:12:05.282327 7659 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:12:05.282408 7659 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:12:05.282490 7653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:05.283150 7653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:05.288726 7653 catalog_manager.cc:1347] Generated new cluster ID: acb51e19733e4132abf983efd1daf33d
I20260924 16:12:05.288815 7653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:05.302796 7653 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:05.303766 7653 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:05.311614 7653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e4bb7bb0624453cbaffb55ee6675853: Generated new TSK 0
I20260924 16:12:05.312333 7653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:12:05.455916 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:05.321214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.1.253.126:38547)
0924 16:12:05.321359 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:05.321367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:05.321537 (+ 170us) server_negotiation.cc:407] Connection header received
0924 16:12:05.321722 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:05.321725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:05.321773 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:05.321858 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:12:05.322148 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.322742 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.323959 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.324158 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.324232 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:05.324574 (+ 342us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:12:05.455255 (+130681us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 16:12:05.455735 (+ 480us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38658: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 16:12:05.456018 7642 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38658: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 16:12:05.456127 7668 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:05.321115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38547 (local address 127.0.0.1:38658)
0924 16:12:05.321440 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:05.321456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:05.321577 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:05.321888 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:05.321892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:05.321900 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:12:05.322032 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:05.322038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.322857 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.322861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:05.323843 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:05.323854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.324275 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.324279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:05.324382 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:05.324412 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:12:05.455603 (+131191us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 16:12:05.455880 (+ 277us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.253.126:38547: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":122,"threads_started":1}
I20260924 16:12:05.457005 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 7572
2026-09-24T16:12:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (407 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (127 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (125 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:05 dist-test-slave-6wvm krb5kdc[7680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:05 dist-test-slave-6wvm krb5kdc[7680](info): set up 2 sockets
Sep 24 16:12:05 dist-test-slave-6wvm krb5kdc[7680](info): commencing operation
krb5kdc: starting...
W20260924 16:12:06.746228 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:12:06 dist-test-slave-6wvm krb5kdc[7680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:12:06.867319 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:35457
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:35457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:06.981256 7693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:06.981599 7693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:06.981690 7693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:06.985847 7693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:06.985953 7693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:06.985992 7693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:06.986032 7693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:06.986071 7693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:06.991899 7693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:35457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:35457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:06.993119 7693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:06.994287 7693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:07.003134 7698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:07.003244 7701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:07.003194 7693 server_base.cc:1034] running on GCE node
W20260924 16:12:07.003141 7699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:07.004109 7693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:07.005250 7693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:07.006453 7693 hybrid_clock.cc:648] HybridClock initialized: now 1790266327006432 us; error 38 us; skew 500 ppm
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:07.015156 7693 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:12:07.016258 7693 webserver.cc:460] Webserver started at http://127.1.253.126:40503/ using document root <none> and password file <none>
I20260924 16:12:07.017150 7693 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:07.017269 7693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:07.017551 7693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:07.019824 7693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "1933904f94874bce990246aa11e56ed1"
format_stamp: "Formatted at 2026-09-24 16:12:07 on dist-test-slave-6wvm"
server_key: "b69a2549d6c03ad2e08219fda6537859"
server_key_iv: "2821b7d2bd3179e98ed7623c4652f40d"
server_key_version: "encryptionkey@0"
I20260924 16:12:07.020458 7693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "1933904f94874bce990246aa11e56ed1"
format_stamp: "Formatted at 2026-09-24 16:12:07 on dist-test-slave-6wvm"
server_key: "b69a2549d6c03ad2e08219fda6537859"
server_key_iv: "2821b7d2bd3179e98ed7623c4652f40d"
server_key_version: "encryptionkey@0"
I20260924 16:12:07.024636 7693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:12:07.027283 7708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:07.028584 7693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:07.028743 7693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "1933904f94874bce990246aa11e56ed1"
format_stamp: "Formatted at 2026-09-24 16:12:07 on dist-test-slave-6wvm"
server_key: "b69a2549d6c03ad2e08219fda6537859"
server_key_iv: "2821b7d2bd3179e98ed7623c4652f40d"
server_key_version: "encryptionkey@0"
I20260924 16:12:07.028862 7693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:07.043840 7693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:07.046633 7693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:07.046852 7693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:07.053912 7693 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:35457
I20260924 16:12:07.053967 7760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:35457 every 8 connection(s)
I20260924 16:12:07.054949 7693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:07.057901 7761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:07.063802 7761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1: Bootstrap starting.
I20260924 16:12:07.064633 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 7693
I20260924 16:12:07.064769 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:07.065096 2037 external_mini_cluster.cc:1428] Setting key 9cb00f63fcea10f8caa833d78c795273
I20260924 16:12:07.067019 7761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:07.067785 7761 log.cc:824] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:07.070003 7761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1: No bootstrap required, opened a new log
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:07.074779 7761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1933904f94874bce990246aa11e56ed1" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35457 } }
I20260924 16:12:07.075016 7761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:07.075088 7761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1933904f94874bce990246aa11e56ed1, State: Initialized, Role: FOLLOWER
I20260924 16:12:07.075701 7761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1933904f94874bce990246aa11e56ed1" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35457 } }
I20260924 16:12:07.075891 7761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:07.075973 7761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:07.076074 7761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:07.077088 7761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1933904f94874bce990246aa11e56ed1" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35457 } }
I20260924 16:12:07.077448 7761 leader_election.cc:302] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1933904f94874bce990246aa11e56ed1; no voters:
I20260924 16:12:07.077806 7761 leader_election.cc:288] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:07.077881 7766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:07.078104 7766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [term 1 LEADER]: Becoming Leader. State: Replica: 1933904f94874bce990246aa11e56ed1, State: Running, Role: LEADER
I20260924 16:12:07.078496 7766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1933904f94874bce990246aa11e56ed1" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35457 } }
I20260924 16:12:07.079201 7761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:07.079850 7768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1933904f94874bce990246aa11e56ed1. Latest consensus state: current_term: 1 leader_uuid: "1933904f94874bce990246aa11e56ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1933904f94874bce990246aa11e56ed1" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35457 } } }
I20260924 16:12:07.080026 7768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:07.080346 7767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1933904f94874bce990246aa11e56ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1933904f94874bce990246aa11e56ed1" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 35457 } } }
I20260924 16:12:07.080520 7767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:07.080554 7772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:07.080822 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:07.066491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.1.253.126:35457)
0924 16:12:07.066935 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:07.066952 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:07.066989 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:12:07.067682 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:07.067701 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:07.068097 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:07.068508 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:07.069375 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.070444 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.071571 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.072028 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.072144 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:07.075245 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:07.075250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:07.075265 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:07.075295 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:07.077200 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.077674 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.077683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.077689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.077817 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.078205 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.078209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.078211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.078770 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:07.078981 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:07.079378 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 16:12:07.079717 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":150,"threads_started":1}
I20260924 16:12:07.083928 7772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:07.090168 7772 catalog_manager.cc:1347] Generated new cluster ID: de9094e8989b4fc8977db6e2f7bacf9d
I20260924 16:12:07.090271 7772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:07.099545 7772 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:07.100805 7772 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:07.111960 7772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1933904f94874bce990246aa11e56ed1: Generated new TSK 0
I20260924 16:12:07.112848 7772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:12:07.144498 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:35457
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:07.259547 7787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:07.259795 7787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:07.259840 7787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:07.263928 7787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:07.263984 7787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:07.264081 7787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:12:07.269232 7787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:35457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:07.270439 7787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:07.271530 7787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:07.281481 7795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:07.281530 7787 server_base.cc:1034] running on GCE node
W20260924 16:12:07.281486 7793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:07.281499 7792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:07.282171 7787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:07.282842 7787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:07.284029 7787 hybrid_clock.cc:648] HybridClock initialized: now 1790266327284014 us; error 29 us; skew 500 ppm
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:07.286890 7787 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:12:07.287781 7787 webserver.cc:460] Webserver started at http://127.1.253.65:32775/ using document root <none> and password file <none>
I20260924 16:12:07.288372 7787 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:07.288467 7787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:07.288719 7787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:07.290982 7787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "4507c46bf7f74646b8116d2b2c7210d1"
format_stamp: "Formatted at 2026-09-24 16:12:07 on dist-test-slave-6wvm"
server_key: "0c73ca79997054d0ca8e701982c45b8d"
server_key_iv: "2d678040970078359d9eb61385fde588"
server_key_version: "encryptionkey@0"
I20260924 16:12:07.291592 7787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "4507c46bf7f74646b8116d2b2c7210d1"
format_stamp: "Formatted at 2026-09-24 16:12:07 on dist-test-slave-6wvm"
server_key: "0c73ca79997054d0ca8e701982c45b8d"
server_key_iv: "2d678040970078359d9eb61385fde588"
server_key_version: "encryptionkey@0"
I20260924 16:12:07.295543 7787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:12:07.298205 7802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:07.299643 7787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:07.299749 7787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "4507c46bf7f74646b8116d2b2c7210d1"
format_stamp: "Formatted at 2026-09-24 16:12:07 on dist-test-slave-6wvm"
server_key: "0c73ca79997054d0ca8e701982c45b8d"
server_key_iv: "2d678040970078359d9eb61385fde588"
server_key_version: "encryptionkey@0"
I20260924 16:12:07.299832 7787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:07.316697 7787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:07.319720 7787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:07.319875 7787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:07.320474 7787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:07.321269 7787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:07.321316 7787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:07.321355 7787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:07.321370 7787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:07.330955 7787 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:33361
I20260924 16:12:07.330984 7915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:33361 every 8 connection(s)
I20260924 16:12:07.331873 7787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:07.341493 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 7787
I20260924 16:12:07.341650 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:07.341940 2037 external_mini_cluster.cc:1428] Setting key 2659e053b35a7efae0a45a33a8ee71a7
I20260924 16:12:07.346360 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:07.333810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:34095 (local address 127.1.253.126:35457)
0924 16:12:07.333972 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:07.333977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:07.334895 (+ 918us) server_negotiation.cc:407] Connection header received
0924 16:12:07.336296 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:07.336298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:07.336346 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:07.336419 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:07.337559 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.338082 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.339234 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.339452 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.339535 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:07.342086 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:07.342089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:07.342091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:07.342114 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:07.343796 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.344327 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.344332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.344333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.344415 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.344781 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.344785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.344787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.344973 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:07.345121 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:07.345784 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 16:12:07.345985 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:12:07.348392 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:07.334288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:35457 (local address 127.1.253.65:34095)
0924 16:12:07.334744 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:07.334786 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:07.336101 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:07.336593 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:07.336601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:07.336948 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:07.337425 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:07.337438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.338188 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.338190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:07.339132 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:07.339141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.339598 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.339601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:07.339738 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:07.340363 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:07.340379 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:07.341908 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:07.343919 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:07.343925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:07.343942 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:07.344219 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:07.344511 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:07.344515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:07.344517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:07.344680 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:07.345088 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:07.345096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:07.345456 (+ 360us) client_negotiation.cc:769] Sending connection context
0924 16:12:07.346554 (+ 1098us) client_negotiation.cc:240] Negotiation successful
0924 16:12:07.346973 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
I20260924 16:12:07.350682 7916 heartbeater.cc:347] Connected to a master server at 127.1.253.126:35457
I20260924 16:12:07.350986 7916 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:07.351719 7916 heartbeater.cc:511] Master 127.1.253.126:35457 requested a full tablet report, sending...
I20260924 16:12:07.353777 7725 ts_manager.cc:194] Registered new tserver with Master: 4507c46bf7f74646b8116d2b2c7210d1 (127.1.253.65:33361)
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:12:07.355685 7725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:34095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:12:07.375682 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:35457
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:07.504392 7921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:07.504670 7921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:07.504748 7921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:07.508882 7921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:07.508950 7921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:07.509011 7921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:12:07.514322 7921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:35457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:07.515496 7921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:07.516651 7921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:07.526675 7927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:07.526937 7926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:07.526965 7929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:07.527079 7921 server_base.cc:1034] running on GCE node
I20260924 16:12:07.527477 7921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:07.528065 7921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:07.529222 7921 hybrid_clock.cc:648] HybridClock initialized: now 1790266327529188 us; error 47 us; skew 500 ppm
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:07.532073 7921 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:12:07.532945 7921 webserver.cc:460] Webserver started at http://127.1.253.66:46141/ using document root <none> and password file <none>
I20260924 16:12:07.533543 7921 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:07.533607 7921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:07.533847 7921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:07.536002 7921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "f59c2b193bff483e898e6b451c3f5fd6"
format_stamp: "Formatted at 2026-09-24 16:12:07 on dist-test-slave-6wvm"
server_key: "24b12b39e1a5c0bf132f9866ac515e76"
server_key_iv: "aa2be61482ea2de248d54ff112d8e9d2"
server_key_version: "encryptionkey@0"
I20260924 16:12:07.536586 7921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "f59c2b193bff483e898e6b451c3f5fd6"
format_stamp: "Formatted at 2026-09-24 16:12:07 on dist-test-slave-6wvm"
server_key: "24b12b39e1a5c0bf132f9866ac515e76"
server_key_iv: "aa2be61482ea2de248d54ff112d8e9d2"
server_key_version: "encryptionkey@0"
I20260924 16:12:07.540354 7921 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 16:12:07.542750 7936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:07.543955 7921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:07.544076 7921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "f59c2b193bff483e898e6b451c3f5fd6"
format_stamp: "Formatted at 2026-09-24 16:12:07 on dist-test-slave-6wvm"
server_key: "24b12b39e1a5c0bf132f9866ac515e76"
server_key_iv: "aa2be61482ea2de248d54ff112d8e9d2"
server_key_version: "encryptionkey@0"
I20260924 16:12:07.544178 7921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:07.560765 7921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:07.564296 7921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:07.564455 7921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:07.565124 7921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:07.565974 7921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:07.566028 7921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:07.566113 7921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:07.566152 7921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:07.575081 7921 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:44227
I20260924 16:12:07.575101 8050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:44227 every 8 connection(s)
I20260924 16:12:07.575999 7921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:07.584077 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 7921
I20260924 16:12:07.584213 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:07.584558 2037 external_mini_cluster.cc:1428] Setting key 0e9b0113cb8fea953905b24c867b745c
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:07.591188 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:07.577898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:41353 (local address 127.1.253.126:35457)
0924 16:12:07.578036 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:07.578040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:07.578970 (+ 930us) server_negotiation.cc:407] Connection header received
0924 16:12:07.580835 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:07.580839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:07.580928 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:07.581026 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:07.582366 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.582896 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.584037 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.584373 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.584524 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:07.587407 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:07.587413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:07.587419 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:07.587455 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:07.588994 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.589585 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.589591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.589593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.589674 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.590026 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.590029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.590031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.590221 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:07.590363 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:07.590899 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 16:12:07.591022 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:12:07.592079 8053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:07.578315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:35457 (local address 127.1.253.66:41353)
0924 16:12:07.578798 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:07.578841 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:07.580598 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:07.581220 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:07.581227 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:07.581669 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:07.582167 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:07.582181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.583004 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.583007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:07.583898 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:07.583907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.584737 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.584740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:07.584885 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:07.585581 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:07.585603 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:07.587236 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:07.589121 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:07.589130 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:07.589147 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:07.589460 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:07.589771 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:07.589775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:07.589777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:07.589918 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:07.590383 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:07.590390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:07.590649 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 16:12:07.590841 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 16:12:07.591202 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":107,"threads_started":1}
I20260924 16:12:07.593492 8051 heartbeater.cc:347] Connected to a master server at 127.1.253.126:35457
I20260924 16:12:07.593763 8051 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:07.594344 8051 heartbeater.cc:511] Master 127.1.253.126:35457 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:12:07.595433 7725 ts_manager.cc:194] Registered new tserver with Master: f59c2b193bff483e898e6b451c3f5fd6 (127.1.253.66:44227)
I20260924 16:12:07.596082 7725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:41353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:12:07.612761 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:35457
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:07.727357 8056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:07.727598 8056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:07.727649 8056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:07.731746 8056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:07.731809 8056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:07.731899 8056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:12:07.737531 8056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:35457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:07.738775 8056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:07.739876 8056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:07.750056 8061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:07.750085 8064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:07.750056 8062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:07.750108 8056 server_base.cc:1034] running on GCE node
I20260924 16:12:07.750967 8056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:07.751684 8056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:07.752872 8056 hybrid_clock.cc:648] HybridClock initialized: now 1790266327752849 us; error 41 us; skew 500 ppm
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:07.755802 8056 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:12:07.756748 8056 webserver.cc:460] Webserver started at http://127.1.253.67:46089/ using document root <none> and password file <none>
I20260924 16:12:07.757335 8056 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:07.757427 8056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:07.757683 8056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:07.760023 8056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "aa1a6afcca0a45d4a4e6cea79ff3c67f"
format_stamp: "Formatted at 2026-09-24 16:12:07 on dist-test-slave-6wvm"
server_key: "ccfd72b90fd8c4948fe31997a5c9fa7d"
server_key_iv: "59e5f902ea2f1c3f3ea6d4931d244899"
server_key_version: "encryptionkey@0"
I20260924 16:12:07.760635 8056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "aa1a6afcca0a45d4a4e6cea79ff3c67f"
format_stamp: "Formatted at 2026-09-24 16:12:07 on dist-test-slave-6wvm"
server_key: "ccfd72b90fd8c4948fe31997a5c9fa7d"
server_key_iv: "59e5f902ea2f1c3f3ea6d4931d244899"
server_key_version: "encryptionkey@0"
I20260924 16:12:07.764470 8056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:12:07.767077 8071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:07.768303 8056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:07.768452 8056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "aa1a6afcca0a45d4a4e6cea79ff3c67f"
format_stamp: "Formatted at 2026-09-24 16:12:07 on dist-test-slave-6wvm"
server_key: "ccfd72b90fd8c4948fe31997a5c9fa7d"
server_key_iv: "59e5f902ea2f1c3f3ea6d4931d244899"
server_key_version: "encryptionkey@0"
I20260924 16:12:07.768587 8056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:07.795996 8056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:07.799712 8056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:07.799930 8056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:07.800688 8056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:07.801574 8056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:07.801647 8056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:07.801720 8056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:07.801771 8056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:07.811424 8056 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:40275
I20260924 16:12:07.811483 8185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:40275 every 8 connection(s)
I20260924 16:12:07.812429 8056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:07.820210 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 8056
I20260924 16:12:07.820366 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:07.820736 2037 external_mini_cluster.cc:1428] Setting key e6d7589325f2eebea5c933bd8fe3d057
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:07.827294 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:07.814444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:45115 (local address 127.1.253.126:35457)
0924 16:12:07.814582 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:07.814586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:07.815541 (+ 955us) server_negotiation.cc:407] Connection header received
0924 16:12:07.817032 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:07.817036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:07.817104 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:07.817222 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:07.818593 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.819145 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.820623 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.820895 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.820987 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:07.824006 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:07.824010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:07.824012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:07.824037 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:07.825339 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.825831 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.825834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.825835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.825934 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.826292 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.826296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.826298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.826505 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:07.826596 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:07.827062 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 16:12:07.827166 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 16:12:07.828125 8188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:07.814864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:35457 (local address 127.1.253.67:45115)
0924 16:12:07.815354 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:07.815400 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:07.816786 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:07.817405 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:07.817413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:07.817838 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:07.818429 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:07.818448 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.819289 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.819294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:07.820485 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:07.820499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.821071 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.821078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:07.821257 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:07.821929 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:07.821946 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:07.823843 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:07.825451 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:07.825456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:07.825468 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:07.825724 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:07.826018 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:07.826020 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:07.826022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:07.826151 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:07.826604 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:07.826609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:07.826839 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 16:12:07.826988 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 16:12:07.827233 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":107,"threads_started":1}
I20260924 16:12:07.829329 8186 heartbeater.cc:347] Connected to a master server at 127.1.253.126:35457
I20260924 16:12:07.829571 8186 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:07.830037 8186 heartbeater.cc:511] Master 127.1.253.126:35457 requested a full tablet report, sending...
I20260924 16:12:07.831105 7725 ts_manager.cc:194] Registered new tserver with Master: aa1a6afcca0a45d4a4e6cea79ff3c67f (127.1.253.67:40275)
I20260924 16:12:07.831729 7725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:45115
I20260924 16:12:07.835028 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/test-admin.keytab.
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:07.851459 2037 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 16:12:07.854404 8193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:07.852879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.1.253.65:33361)
0924 16:12:07.853232 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:07.853239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:07.853266 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:12:07.853336 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:07.853342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:07.853536 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:07.853650 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:07.854196 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57550: BlockingRecv error: recv got EOF from 127.0.0.1:57550 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":122,"threads_started":1}
W20260924 16:12:07.854827 2037 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/krb5cc' of type FILE
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:07.855883 2037 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:07.856809 2037 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.65@KRBTEST.COM
I20260924 16:12:07.867229 8193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:07.857800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.1.253.65:33361)
0924 16:12:07.857963 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:07.857968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:07.858079 (+ 111us) server_negotiation.cc:407] Connection header received
0924 16:12:07.858286 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:07.858289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:07.858336 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:07.858432 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:07.859146 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.859910 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.860784 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.861198 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.861323 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:07.863727 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:07.863738 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:07.863754 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:07.863782 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:07.865292 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.865706 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.865712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.865713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.865797 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.866075 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.866087 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.866088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.866464 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:07.866661 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:07.866913 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 16:12:07.867090 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 16:12:07.871480 8193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:07.870518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.1.253.65:33361)
0924 16:12:07.870670 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:07.870674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:07.870720 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:12:07.870827 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:07.870830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:07.870868 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:07.870935 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:07.871375 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57554: BlockingRecv error: recv got EOF from 127.0.0.1:57554 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 16:12:07.871825 2037 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/krb5cc' of type FILE
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:07.873041 2037 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:07.873993 2037 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.65@KRBTEST.COM
I20260924 16:12:07.883085 8193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:07.875027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.1.253.65:33361)
0924 16:12:07.875187 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:07.875191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:07.875314 (+ 123us) server_negotiation.cc:407] Connection header received
0924 16:12:07.875472 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:07.875475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:07.875520 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:07.875616 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:07.876139 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.876680 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.877504 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.877760 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.877871 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:07.880238 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:07.880241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:07.880243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:07.880267 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:07.881635 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.882028 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.882031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.882032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.882109 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.882381 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.882383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.882384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.882541 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:07.882656 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:07.882851 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 16:12:07.882956 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 16:12:07.885831 8193 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:07.884955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.1.253.65:33361)
0924 16:12:07.885104 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:07.885107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:07.885159 (+ 52us) server_negotiation.cc:407] Connection header received
0924 16:12:07.885254 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:07.885256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:07.885293 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:07.885357 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:07.885735 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57572: BlockingRecv error: recv got EOF from 127.0.0.1:57572 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 16:12:07.886268 2037 init.cc:244] Error while opening credential cache '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266297063660-2037-0/krb5kdc/krb5cc)
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:07.887317 2037 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:07.888230 2037 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[7680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.65@KRBTEST.COM
I20260924 16:12:07.896224 8193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:07.889103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.1.253.65:33361)
0924 16:12:07.889229 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:07.889232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:07.889244 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:12:07.889352 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:07.889355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:07.889391 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:07.889454 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:07.889895 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.890438 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.891250 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.891448 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.891518 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:07.893627 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:07.893631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:07.893633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:07.893653 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:07.894899 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.895272 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.895275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.895276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.895344 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.895572 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.895574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.895575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.895722 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:07.895836 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:07.896023 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 16:12:07.896110 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:12:07.897413 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 7787
I20260924 16:12:07.904830 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 7921
I20260924 16:12:07.911139 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 8056
I20260924 16:12:07.917629 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 7693
2026-09-24T16:12:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2208 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[8206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[8206](info): set up 2 sockets
Sep 24 16:12:07 dist-test-slave-6wvm krb5kdc[8206](info): commencing operation
krb5kdc: starting...
W20260924 16:12:08.970156 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:12:09 dist-test-slave-6wvm krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.126.keytab.
Entry for principal oryx/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.126.keytab.
I20260924 16:12:09.096982 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:33025
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.126.keytab
--principal=oryx/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:33025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:09.209448 8219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:09.209714 8219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:09.209754 8219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:09.213728 8219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:09.213785 8219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:09.213804 8219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:09.213820 8219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:09.213835 8219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:09.219105 8219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:33025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.126.keytab
--principal=oryx/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:09.220269 8219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:09.221369 8219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:09.229948 8225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:09.229986 8219 server_base.cc:1034] running on GCE node
W20260924 16:12:09.229948 8224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:09.230007 8227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:09.230827 8219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:09.231938 8219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:09.233137 8219 hybrid_clock.cc:648] HybridClock initialized: now 1790266329233129 us; error 32 us; skew 500 ppm
Sep 24 16:12:09 dist-test-slave-6wvm krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:09.235883 8219 init.cc:377] Logged in from keytab as oryx/127.1.253.126@KRBTEST.COM (short username oryx)
I20260924 16:12:09.236864 8219 webserver.cc:460] Webserver started at http://127.1.253.126:42389/ using document root <none> and password file <none>
I20260924 16:12:09.237440 8219 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:09.237541 8219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:09.237778 8219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:09.239907 8219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "519c19cb2bb7494d9f685cfa7766ea10"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "6cfa7976584f58ab91f65649c4a5aa88"
server_key_iv: "6b274234c5014fde07c4bb7ce18d4b15"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.240494 8219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "519c19cb2bb7494d9f685cfa7766ea10"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "6cfa7976584f58ab91f65649c4a5aa88"
server_key_iv: "6b274234c5014fde07c4bb7ce18d4b15"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.244182 8219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260924 16:12:09.246507 8234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:09.247673 8219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:12:09.247815 8219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "519c19cb2bb7494d9f685cfa7766ea10"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "6cfa7976584f58ab91f65649c4a5aa88"
server_key_iv: "6b274234c5014fde07c4bb7ce18d4b15"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.247921 8219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:09.281266 8219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:09.284757 8219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:09.284987 8219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:09.292943 8219 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:33025
I20260924 16:12:09.293247 8286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:33025 every 8 connection(s)
I20260924 16:12:09.294068 8219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:09.297048 8287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:09.302484 8287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10: Bootstrap starting.
I20260924 16:12:09.304032 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 8219
I20260924 16:12:09.304204 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:09.304494 2037 external_mini_cluster.cc:1428] Setting key 46d0535c72657281bbdc7c63ee8f80a2
I20260924 16:12:09.305020 8287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:09.305896 8287 log.cc:824] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:09.307775 8287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10: No bootstrap required, opened a new log
I20260924 16:12:09.311237 8287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } }
I20260924 16:12:09.311479 8287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:09.311539 8287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519c19cb2bb7494d9f685cfa7766ea10, State: Initialized, Role: FOLLOWER
I20260924 16:12:09.311993 8287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } }
I20260924 16:12:09.312158 8287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:09.312248 8287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:09.312368 8287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:09.313380 8287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } }
Sep 24 16:12:09 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
I20260924 16:12:09.313753 8287 leader_election.cc:302] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 519c19cb2bb7494d9f685cfa7766ea10; no voters:
I20260924 16:12:09.314092 8287 leader_election.cc:288] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:09.314221 8292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:09.314499 8292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 1 LEADER]: Becoming Leader. State: Replica: 519c19cb2bb7494d9f685cfa7766ea10, State: Running, Role: LEADER
I20260924 16:12:09.314859 8292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } }
I20260924 16:12:09.315598 8287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:09.316623 8294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519c19cb2bb7494d9f685cfa7766ea10. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } }
I20260924 16:12:09.316756 8294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:09.317175 8293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } }
I20260924 16:12:09.317322 8293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:09.317663 8301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:09.320312 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:09.305701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35776 (local address 127.1.253.126:33025)
0924 16:12:09.306290 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:09.306312 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:09.306355 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:12:09.307045 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:09.307062 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:09.307390 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:09.307736 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:09.308581 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.309553 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.310533 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.310995 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.311173 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:09.314123 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:09.314129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:09.314147 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:09.314189 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:09.316740 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:09.317321 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:09.317330 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:09.317338 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:09.317496 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:09.317835 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:09.317838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:09.317840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:09.318390 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:09.318599 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:09.319021 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 16:12:09.319329 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":155,"threads_started":1}
I20260924 16:12:09.320645 8301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:09.326632 8301 catalog_manager.cc:1347] Generated new cluster ID: 6c776d6ba04147bdbca859a4c9754ca3
I20260924 16:12:09.326696 8301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:09.343333 8301 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:09.344535 8301 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:09.350026 8301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10: Generated new TSK 0
I20260924 16:12:09.350764 8301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab.
Entry for principal oryx/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab.
I20260924 16:12:09.380950 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:33025
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab
--principal=oryx/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:09.496179 8313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:09.496450 8313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:09.496498 8313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:09.500618 8313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:09.500684 8313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:09.500744 8313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:12:09.505981 8313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab
--principal=oryx/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:09.507266 8313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:09.508319 8313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:09.518630 8321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:09.518674 8319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:09.518646 8318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:09.518962 8313 server_base.cc:1034] running on GCE node
I20260924 16:12:09.519470 8313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:09.520213 8313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:09.521407 8313 hybrid_clock.cc:648] HybridClock initialized: now 1790266329521371 us; error 59 us; skew 500 ppm
Sep 24 16:12:09 dist-test-slave-6wvm krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:09.524302 8313 init.cc:377] Logged in from keytab as oryx/127.1.253.65@KRBTEST.COM (short username oryx)
I20260924 16:12:09.525293 8313 webserver.cc:460] Webserver started at http://127.1.253.65:38515/ using document root <none> and password file <none>
I20260924 16:12:09.525905 8313 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:09.526000 8313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:09.526305 8313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:09.528522 8313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "e2a33489ba884ef0b2eaf21e7897618b"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "77b3704a110746c8612bf9a914924308"
server_key_iv: "b702c2e7d9605e3ba7932a34c91ab51c"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.529131 8313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "e2a33489ba884ef0b2eaf21e7897618b"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "77b3704a110746c8612bf9a914924308"
server_key_iv: "b702c2e7d9605e3ba7932a34c91ab51c"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.533133 8313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 16:12:09.535739 8328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:09.536990 8313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:12:09.537139 8313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "e2a33489ba884ef0b2eaf21e7897618b"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "77b3704a110746c8612bf9a914924308"
server_key_iv: "b702c2e7d9605e3ba7932a34c91ab51c"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.537272 8313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:09.554333 8313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:09.557314 8313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:09.557468 8313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:09.558084 8313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:09.558946 8313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:09.558992 8313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:09.559031 8313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:09.559046 8313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:09.567916 8313 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:40341
I20260924 16:12:09.567935 8441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:40341 every 8 connection(s)
I20260924 16:12:09.568848 8313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:09.578163 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 8313
I20260924 16:12:09.578328 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:09.578613 2037 external_mini_cluster.cc:1428] Setting key 5d995a603b2d6ce24b01d3833eb86922
Sep 24 16:12:09 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
I20260924 16:12:09.583431 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:09.570777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:34125 (local address 127.1.253.126:33025)
0924 16:12:09.570924 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:09.570928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:09.571807 (+ 879us) server_negotiation.cc:407] Connection header received
0924 16:12:09.573126 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:09.573129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:09.573180 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:09.573257 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:09.574577 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.575122 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.576250 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.576478 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.576562 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:09.579354 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:09.579357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:09.579359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:09.579410 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:09.580948 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:09.581613 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:09.581616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:09.581617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:09.581694 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:09.582110 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:09.582115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:09.582118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:09.582417 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:09.582526 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:09.583047 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 16:12:09.583211 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:12:09.584234 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:09.571185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.65:34125)
0924 16:12:09.571643 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:09.571686 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:09.572915 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:09.573442 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:09.573448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:09.573819 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:09.574425 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:09.574444 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.575235 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.575238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:09.576117 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:09.576126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.576618 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.576620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:09.576767 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:09.577418 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:09.577436 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:09.579192 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:09.581076 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:09.581086 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:09.581108 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:09.581509 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:09.581786 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:09.581792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:09.581797 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:09.582000 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:09.582525 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:09.582531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:09.582795 (+ 264us) client_negotiation.cc:769] Sending connection context
0924 16:12:09.582962 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 16:12:09.583241 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":106,"threads_started":1}
I20260924 16:12:09.585736 8442 heartbeater.cc:347] Connected to a master server at 127.1.253.126:33025
I20260924 16:12:09.586043 8442 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:09.586812 8442 heartbeater.cc:511] Master 127.1.253.126:33025 requested a full tablet report, sending...
I20260924 16:12:09.588587 8251 ts_manager.cc:194] Registered new tserver with Master: e2a33489ba884ef0b2eaf21e7897618b (127.1.253.65:40341)
WARNING: no policy specified for oryx/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:12:09.589833 8251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:34125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab.
Entry for principal oryx/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab.
I20260924 16:12:09.607352 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:33025
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab
--principal=oryx/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:09.720090 8447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:09.720386 8447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:09.720428 8447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:09.724457 8447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:09.724520 8447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:09.724579 8447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:12:09.729861 8447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab
--principal=oryx/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:09.730980 8447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:09.731989 8447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:09.741859 8452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:09.741812 8453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:09.742007 8447 server_base.cc:1034] running on GCE node
W20260924 16:12:09.741812 8455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:09.742601 8447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:09.743273 8447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:09.744437 8447 hybrid_clock.cc:648] HybridClock initialized: now 1790266329744415 us; error 32 us; skew 500 ppm
Sep 24 16:12:09 dist-test-slave-6wvm krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:09.747360 8447 init.cc:377] Logged in from keytab as oryx/127.1.253.66@KRBTEST.COM (short username oryx)
I20260924 16:12:09.748190 8447 webserver.cc:460] Webserver started at http://127.1.253.66:44447/ using document root <none> and password file <none>
I20260924 16:12:09.748713 8447 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:09.748769 8447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:09.748931 8447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:09.751082 8447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "edc542371ed34c6fa9e7153ad00c242d"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "714d63fdd03532779ff66c39547d5907"
server_key_iv: "ff5e532427dd1bf50051a666a81c7c55"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.751653 8447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "edc542371ed34c6fa9e7153ad00c242d"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "714d63fdd03532779ff66c39547d5907"
server_key_iv: "ff5e532427dd1bf50051a666a81c7c55"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.755450 8447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:12:09.757851 8462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:09.759176 8447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:09.759311 8447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "edc542371ed34c6fa9e7153ad00c242d"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "714d63fdd03532779ff66c39547d5907"
server_key_iv: "ff5e532427dd1bf50051a666a81c7c55"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.759419 8447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:09.778714 8447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:09.782106 8447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:09.782367 8447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:09.782989 8447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:09.783825 8447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:09.783895 8447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:09.783975 8447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:09.784027 8447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:09.793017 8447 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:35059
I20260924 16:12:09.793044 8575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:35059 every 8 connection(s)
I20260924 16:12:09.793906 8447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
Sep 24 16:12:09 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
I20260924 16:12:09.804111 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 8447
I20260924 16:12:09.804253 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:09.804566 2037 external_mini_cluster.cc:1428] Setting key 5b6749d7fa1f185db5dc46137e57732d
I20260924 16:12:09.808677 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:09.795842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:54737 (local address 127.1.253.126:33025)
0924 16:12:09.795984 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:09.795988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:09.796805 (+ 817us) server_negotiation.cc:407] Connection header received
0924 16:12:09.798212 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:09.798215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:09.798300 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:09.798412 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:09.799609 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.800148 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.801338 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.801570 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.801688 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:09.804286 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:09.804289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:09.804292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:09.804325 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:09.806142 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:09.806804 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:09.806810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:09.806815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:09.806916 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:09.807323 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:09.807326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:09.807327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:09.807526 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:09.807603 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:09.808286 (+ 683us) server_negotiation.cc:299] Negotiation successful
0924 16:12:09.808434 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:12:09.809464 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:09.796182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.66:54737)
0924 16:12:09.796639 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:09.796690 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:09.797979 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:09.798597 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:09.798605 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:09.798971 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:09.799456 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:09.799470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.800305 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.800314 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:09.801205 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:09.801214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.801740 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.801744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:09.801912 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:09.802614 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:09.802630 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:09.804109 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:09.806302 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:09.806309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:09.806325 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:09.806691 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:09.807014 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:09.807020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:09.807025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:09.807224 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:09.807632 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:09.807641 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:09.807990 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 16:12:09.808225 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 16:12:09.808558 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":102,"threads_started":1}
I20260924 16:12:09.810716 8576 heartbeater.cc:347] Connected to a master server at 127.1.253.126:33025
I20260924 16:12:09.810985 8576 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:09.811554 8576 heartbeater.cc:511] Master 127.1.253.126:33025 requested a full tablet report, sending...
I20260924 16:12:09.812677 8251 ts_manager.cc:194] Registered new tserver with Master: edc542371ed34c6fa9e7153ad00c242d (127.1.253.66:35059)
I20260924 16:12:09.813270 8251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:54737
WARNING: no policy specified for oryx/127.1.253.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab.
Entry for principal oryx/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab.
I20260924 16:12:09.832388 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:33025
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab
--principal=oryx/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:09.946255 8581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:09.946554 8581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:09.946630 8581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:09.950589 8581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:09.950678 8581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:09.950767 8581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:12:09.956756 8581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab
--principal=oryx/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:09.957919 8581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:09.959036 8581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:09.969334 8586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:09.969344 8589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:09.969398 8581 server_base.cc:1034] running on GCE node
W20260924 16:12:09.969408 8587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:09.969970 8581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:09.970714 8581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:09.971871 8581 hybrid_clock.cc:648] HybridClock initialized: now 1790266329971833 us; error 49 us; skew 500 ppm
Sep 24 16:12:09 dist-test-slave-6wvm krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:09.974747 8581 init.cc:377] Logged in from keytab as oryx/127.1.253.67@KRBTEST.COM (short username oryx)
I20260924 16:12:09.975618 8581 webserver.cc:460] Webserver started at http://127.1.253.67:45517/ using document root <none> and password file <none>
I20260924 16:12:09.976195 8581 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:09.976256 8581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:09.976497 8581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:09.978685 8581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "dff39b9cdeb04b469a97c979003e5cf1"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "67d27125c9f1b9597a219e76ff7d444c"
server_key_iv: "5705f4365375a31a3c833a65f5115eca"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.979259 8581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "dff39b9cdeb04b469a97c979003e5cf1"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "67d27125c9f1b9597a219e76ff7d444c"
server_key_iv: "5705f4365375a31a3c833a65f5115eca"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.982990 8581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:09.985342 8596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:09.986706 8581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:09.986838 8581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "dff39b9cdeb04b469a97c979003e5cf1"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "67d27125c9f1b9597a219e76ff7d444c"
server_key_iv: "5705f4365375a31a3c833a65f5115eca"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.986950 8581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:09.996403 8581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:09.999713 8581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:09.999914 8581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:10.000517 8581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:10.001351 8581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:10.001422 8581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.001488 8581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:10.001539 8581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.010370 8581 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:35207
I20260924 16:12:10.010443 8709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:35207 every 8 connection(s)
I20260924 16:12:10.011288 8581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:10.019512 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 8581
I20260924 16:12:10.019649 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:10.019997 2037 external_mini_cluster.cc:1428] Setting key 4df85b0fe3db9373500bb45cd5576e66
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
I20260924 16:12:10.026151 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.013194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:52153 (local address 127.1.253.126:33025)
0924 16:12:10.013355 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.013359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.014362 (+ 1003us) server_negotiation.cc:407] Connection header received
0924 16:12:10.015857 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.015861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.015939 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.016025 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.017392 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.018178 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.019528 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.019863 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.020011 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.022939 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.022942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.022944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.022970 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.024298 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.024763 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.024766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.024767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.024837 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.025221 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.025223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.025224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.025388 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.025462 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.025930 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.026025 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:12:10.027103 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.013580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.67:52153)
0924 16:12:10.014081 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.014125 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.015615 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.016226 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.016234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.016583 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.017228 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.017244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.018526 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.018529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.019390 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.019399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.020035 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.020039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.020237 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.020961 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.020979 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.022762 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.024403 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.024408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.024420 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.024679 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.024957 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.024960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.024962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.025120 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.025483 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.025488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.025718 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.025886 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.026140 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":100,"threads_started":1}
I20260924 16:12:10.028249 8710 heartbeater.cc:347] Connected to a master server at 127.1.253.126:33025
I20260924 16:12:10.028524 8710 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:10.029050 8710 heartbeater.cc:511] Master 127.1.253.126:33025 requested a full tablet report, sending...
I20260924 16:12:10.030119 8251 ts_manager.cc:194] Registered new tserver with Master: dff39b9cdeb04b469a97c979003e5cf1 (127.1.253.67:35207)
I20260924 16:12:10.030733 8251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:52153
I20260924 16:12:10.034610 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.253.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.125.keytab.
Entry for principal oryx/127.1.253.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.125.keytab.
I20260924 16:12:10.063838 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.125:35437
--webserver_interface=127.1.253.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.125.keytab
--principal=oryx/127.1.253.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:33025,127.1.253.125:35437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:10.183879 8715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:10.184185 8715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:10.184270 8715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:10.188239 8715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:10.188324 8715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:10.188364 8715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:10.188400 8715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:10.188438 8715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:10.193982 8715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:33025,127.1.253.125:35437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.125.keytab
--principal=oryx/127.1.253.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.125:35437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.253.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
W20260924 16:12:10.194274 8715 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.253.126:33025,127.1.253.125:35437'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 16:12:10.195226 8715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:10.196333 8715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:10.205484 8723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:10.205507 8720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:10.205570 8721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:10.205611 8715 server_base.cc:1034] running on GCE node
I20260924 16:12:10.206470 8715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:10.207608 8715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:10.208782 8715 hybrid_clock.cc:648] HybridClock initialized: now 1790266330208747 us; error 48 us; skew 500 ppm
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:10.211941 8715 init.cc:377] Logged in from keytab as oryx/127.1.253.125@KRBTEST.COM (short username oryx)
I20260924 16:12:10.212882 8715 webserver.cc:460] Webserver started at http://127.1.253.125:37955/ using document root <none> and password file <none>
I20260924 16:12:10.213452 8715 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:10.213516 8715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:10.213696 8715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:10.216077 8715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data/instance:
uuid: "b7b93ed626234c58a2920ac0fe0dd3a7"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-6wvm"
server_key: "0eea89bb554f9b5088bb4036ec3062e5"
server_key_iv: "3bd947d638ef35e4b18bb08c82f47af8"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.216765 8715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/wal/instance:
uuid: "b7b93ed626234c58a2920ac0fe0dd3a7"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-6wvm"
server_key: "0eea89bb554f9b5088bb4036ec3062e5"
server_key_iv: "3bd947d638ef35e4b18bb08c82f47af8"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.220839 8715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 16:12:10.223440 8730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.224735 8715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:12:10.224885 8715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/wal
uuid: "b7b93ed626234c58a2920ac0fe0dd3a7"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-6wvm"
server_key: "0eea89bb554f9b5088bb4036ec3062e5"
server_key_iv: "3bd947d638ef35e4b18bb08c82f47af8"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.225008 8715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:10.244454 8715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:10.248057 8715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:10.248276 8715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:10.256072 8715 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.125:35437
I20260924 16:12:10.256354 8782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.125:35437 every 8 connection(s)
I20260924 16:12:10.257177 8715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
I20260924 16:12:10.260165 8783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:10.263963 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 8715
I20260924 16:12:10.264245 8783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.125@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.253.125@KRBTEST.COM
I20260924 16:12:10.277664 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.266286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35790 (local address 127.1.253.126:33025)
0924 16:12:10.266417 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.266421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.266707 (+ 286us) server_negotiation.cc:407] Connection header received
0924 16:12:10.267283 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.267285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.267332 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.267402 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.268448 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.268978 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.270200 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.270463 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.270539 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.273313 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.273318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.273321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.273352 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.275228 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.275843 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.275845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.275847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.275947 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.276354 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.276359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.276361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.276654 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.276775 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.277328 (+ 553us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.277461 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:12:10.278663 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.266107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:35790)
0924 16:12:10.266545 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.266600 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.267055 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.267462 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.267473 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.267957 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.268285 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.268301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.269110 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.269114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.270070 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.270079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.270599 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.270603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.270782 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.271497 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.271524 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.273163 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.275338 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.275345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.275357 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.275730 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.276070 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.276076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.276078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.276240 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.276786 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.276795 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.277075 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.277264 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.277650 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":135,"threads_started":1}
I20260924 16:12:10.278664 8786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.265552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.1.253.125:35437)
0924 16:12:10.265969 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.265991 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.266050 (+ 59us) server_negotiation.cc:407] Connection header received
0924 16:12:10.267055 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.267080 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.267340 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.267529 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.268231 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.269415 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.270307 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.270775 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.270881 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.273858 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.273863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.273874 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.273894 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.275928 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.276352 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.276358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.276362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.276439 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.276735 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.276739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.276741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.277052 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.277272 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.277496 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.277751 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":270,"thread_start_us":146,"threads_started":1}
I20260924 16:12:10.280253 8783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.125@KRBTEST.COM for oryx/127.1.253.125@KRBTEST.COM
I20260924 16:12:10.287974 8786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.280872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.1.253.125:35437)
0924 16:12:10.281187 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.281191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.281201 (+ 10us) server_negotiation.cc:407] Connection header received
0924 16:12:10.281228 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.281230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.281260 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.281321 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.281726 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.282203 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.282996 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.283217 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.283289 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.285169 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.285172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.285173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.285210 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.286582 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.286957 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.286959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.286960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.287067 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.287312 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.287314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.287315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.287458 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.287530 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.287759 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.287856 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260924 16:12:10.288008 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.280803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:35437 (local address 127.0.0.1:59508)
0924 16:12:10.280926 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.280950 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.281016 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.281357 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.281360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.281527 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.281625 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.281631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.282351 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.282354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.282880 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.282889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.283310 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.283312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.283385 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.283827 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.283839 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.285015 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.286680 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.286682 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.286684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.286868 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.287150 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.287152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.287154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.287229 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.287541 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.287544 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.287617 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.287707 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.287801 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 16:12:10.291229 8783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: Bootstrap starting.
I20260924 16:12:10.293591 8783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:10.294407 8783 log.cc:824] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:10.296474 8783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: No bootstrap required, opened a new log
I20260924 16:12:10.299526 8783 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } }
I20260924 16:12:10.299743 8783 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:10.299826 8783 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b93ed626234c58a2920ac0fe0dd3a7, State: Initialized, Role: FOLLOWER
I20260924 16:12:10.300283 8783 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } }
I20260924 16:12:10.300769 8789 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } } }
I20260924 16:12:10.300890 8789 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:12:10.301542 8783 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:12:10.307505 8800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:12:10.307593 8800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:12:10.308810 8715 master_runner.cc:428] Detected that this master b7b93ed626234c58a2920ac0fe0dd3a7 is joining an existing cluster
I20260924 16:12:10.308878 8715 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.253.125:35437
I20260924 16:12:10.319604 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.310995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:35437 (local address 127.0.0.1:59524)
0924 16:12:10.311393 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.311405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.311544 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.311856 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.311858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.312088 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.312197 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.312203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.313212 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.313215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.313795 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.313803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.314228 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.314231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.314353 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.315124 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.315135 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.315707 (+ 572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.317649 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.317653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.317662 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.317938 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.318300 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.318305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.318309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.318395 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.318982 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.318985 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.319074 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.319190 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.319329 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":94,"threads_started":1}
I20260924 16:12:10.319605 8786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.311243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.1.253.125:35437)
0924 16:12:10.311362 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.311366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.311490 (+ 124us) server_negotiation.cc:407] Connection header received
0924 16:12:10.311631 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.311633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.311676 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.311726 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.312307 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.313072 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.313910 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.314121 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.314229 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.315841 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.315845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.315847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.315866 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.317521 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.318078 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.318082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.318084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.318184 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.318518 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.318524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.318528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.318875 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.319000 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.319277 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.319401 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:12:10.319851 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.311058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.1.253.126:33025)
0924 16:12:10.311191 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.311196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.311479 (+ 283us) server_negotiation.cc:407] Connection header received
0924 16:12:10.311624 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.311627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.311676 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.311750 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.312365 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.313138 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.314405 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.314721 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.314837 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.316003 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.316007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.316010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.316041 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.317892 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.318548 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.318551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.318553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.318636 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.318918 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.318922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.318924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.319130 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.319222 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.319541 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.319679 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 16:12:10.320338 8806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.310958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:35794)
0924 16:12:10.311379 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.311395 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.311519 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.311795 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.311799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.312113 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.312251 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.312260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.313258 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.313262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.314293 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.314305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.314839 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.314843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.314932 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.315384 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.315398 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.315860 (+ 462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.318193 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.318195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.318197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.318462 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.318735 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.318739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.318741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.318825 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.319264 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.319267 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.319354 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.320032 (+ 678us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.320164 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":99,"threads_started":1}
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.126@KRBTEST.COM for oryx/127.1.253.125@KRBTEST.COM
I20260924 16:12:10.335323 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.324924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:35437 (local address 127.0.0.1:59530)
0924 16:12:10.325769 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.325823 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.325964 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.326305 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.326312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.326806 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.326962 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.326970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.327636 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.327640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.328740 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.328751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.329180 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.329184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.329316 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.329822 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.329838 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.331416 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.333372 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.333384 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.333393 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.333710 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.334028 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.334032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.334034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.334129 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.334643 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.334650 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.334778 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.334928 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.335127 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":87,"threads_started":1}
I20260924 16:12:10.335966 8786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.324953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.1.253.125:35437)
0924 16:12:10.325078 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.325081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.325900 (+ 819us) server_negotiation.cc:407] Connection header received
0924 16:12:10.326090 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.326092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.326134 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.326199 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.327075 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.327530 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.328851 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.329048 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.329161 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.331560 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.331565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.331568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.331596 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.333249 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.333818 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.333822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.333824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.333908 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.334269 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.334273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.334275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.334491 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.334584 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.335701 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.335812 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:12:10.337931 8251 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.253.125:35437: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7"
member_type: NON_VOTER
last_known_addr {
host: "127.1.253.125"
port: 35437
}
attrs {
promote: true
}
}
dest_uuid: "519c19cb2bb7494d9f685cfa7766ea10"
cas_config_opid_index: -1
I20260924 16:12:10.338716 8251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.1.253.124@KRBTEST.COM; defaulting to no policy
I20260924 16:12:10.340751 8810 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b7b93ed626234c58a2920ac0fe0dd3a7 (127.1.253.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true } } }
I20260924 16:12:10.341392 8757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:10.342948 8810 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.253.125:35437
I20260924 16:12:10.342988 8757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 FOLLOWER]: Refusing update from remote peer 519c19cb2bb7494d9f685cfa7766ea10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.124@KRBTEST.COM" created.
I20260924 16:12:10.343434 8715 master.cc:537] Master@127.1.253.125:35437 shutting down...
I20260924 16:12:10.343673 8810 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:10.345340 8811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true } } }
I20260924 16:12:10.345484 8811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: This master's current role is: LEADER
W20260924 16:12:10.345621 8236 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 16:12:10.346637 8236 consensus_peers.cc:487] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 -> Peer b7b93ed626234c58a2920ac0fe0dd3a7 (127.1.253.125:35437): Couldn't send request to peer b7b93ed626234c58a2920ac0fe0dd3a7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 16:12:10.351403 8715 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:10.351702 8715 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:10.351836 8715 tablet_replica.cc:330] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: stopping tablet replica
I20260924 16:12:10.357077 8715 master.cc:552] Master@127.1.253.125:35437 shutdown complete.
I20260924 16:12:10.357141 8715 master_runner.cc:305] Clearing existing system tablet
I20260924 16:12:10.357388 8715 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:10.357460 8715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:10.358615 8715 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 16:12:10.359485 8816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.359684 8715 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:10.360097 8715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:10.360142 8715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/wal
uuid: "b7b93ed626234c58a2920ac0fe0dd3a7"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-6wvm"
server_key: "0eea89bb554f9b5088bb4036ec3062e5"
server_key_iv: "3bd947d638ef35e4b18bb08c82f47af8"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.362879 8715 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:10.365483 8715 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 16:12:10.365594 8715 log.cc:1196] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 16:12:10.365906 8715 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: Deleting consensus metadata
I20260924 16:12:10.366101 8715 master_runner.cc:315] Copying system tablet from 127.1.253.126:33025
I20260924 16:12:10.367502 8715 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: tablet copy: Beginning tablet copy session from remote peer at address 127.1.253.126:33025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.124.keytab.
Entry for principal oryx/127.1.253.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.124.keytab.
I20260924 16:12:10.369884 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.124:40835
--webserver_interface=127.1.253.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.124.keytab
--principal=oryx/127.1.253.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:33025,127.1.253.125:35437,127.1.253.124:40835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:12:10.379015 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.368293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.1.253.126:33025)
0924 16:12:10.368415 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.368419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.368486 (+ 67us) server_negotiation.cc:407] Connection header received
0924 16:12:10.368644 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.368646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.368729 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.368836 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.369444 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.370343 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.371693 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.372072 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.372156 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.375014 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.375018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.375022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.375075 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.376946 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.377491 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.377495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.377497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.377586 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.377905 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.377908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.377910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.378148 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.378226 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.378687 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.378826 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 16:12:10.379891 8822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.368084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:35804)
0924 16:12:10.368390 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.368408 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.368550 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.368872 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.368876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.369164 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.369293 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.369301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.370487 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.370492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.371534 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.371545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.372330 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.372334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.372430 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.374046 (+ 1616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.374064 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.374786 (+ 722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.377084 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.377088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.377090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.377354 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.377707 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.377710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.377713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.377801 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.378356 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.378360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.378480 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.379505 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.379675 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":112,"threads_started":1}
I20260924 16:12:10.380868 8271 tablet_copy_service.cc:140] P 519c19cb2bb7494d9f685cfa7766ea10: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b7b93ed626234c58a2920ac0fe0dd3a7 ({username='oryx', principal='oryx/127.1.253.125@KRBTEST.COM'} at 127.0.0.1:35804)
I20260924 16:12:10.381102 8271 tablet_copy_service.cc:161] P 519c19cb2bb7494d9f685cfa7766ea10: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b7b93ed626234c58a2920ac0fe0dd3a7 at {username='oryx', principal='oryx/127.1.253.125@KRBTEST.COM'} at 127.0.0.1:35804: session id = b7b93ed626234c58a2920ac0fe0dd3a7-00000000000000000000000000000000
I20260924 16:12:10.383390 8271 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 16:12:10.385378 8715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 16:12:10.388087 8715 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: tablet copy: Starting download of 0 data blocks...
I20260924 16:12:10.388294 8715 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: tablet copy: Starting download of 1 WAL segments...
I20260924 16:12:10.390295 8715 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 16:12:10.392256 8271 tablet_copy_service.cc:342] P 519c19cb2bb7494d9f685cfa7766ea10: Request end of tablet copy session b7b93ed626234c58a2920ac0fe0dd3a7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.253.125@KRBTEST.COM'} at 127.0.0.1:35804
I20260924 16:12:10.392379 8271 tablet_copy_service.cc:433] P 519c19cb2bb7494d9f685cfa7766ea10: ending tablet copy session b7b93ed626234c58a2920ac0fe0dd3a7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b7b93ed626234c58a2920ac0fe0dd3a7
I20260924 16:12:10.400731 8715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:10.402520 8824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:10.402773 8827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:10.402976 8825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:10.403087 8715 server_base.cc:1034] running on GCE node
I20260924 16:12:10.403331 8715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:10.403662 8715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:10.404814 8715 hybrid_clock.cc:648] HybridClock initialized: now 1790266330404791 us; error 40 us; skew 500 ppm
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:10.406565 8715 init.cc:377] Logged in from keytab as oryx/127.1.253.125@KRBTEST.COM (short username oryx)
I20260924 16:12:10.407059 8715 webserver.cc:460] Webserver started at http://127.1.253.125:32997/ using document root <none> and password file <none>
I20260924 16:12:10.407266 8715 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:10.407312 8715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:10.408743 8715 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:10.409871 8834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.410156 8715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:10.410334 8715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/wal
uuid: "b7b93ed626234c58a2920ac0fe0dd3a7"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-6wvm"
server_key: "0eea89bb554f9b5088bb4036ec3062e5"
server_key_iv: "3bd947d638ef35e4b18bb08c82f47af8"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.410506 8715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:10.440929 8715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:10.443365 8715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:10.448841 8715 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.125:35437
I20260924 16:12:10.449630 8715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
I20260924 16:12:10.450186 8886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.125:35437 every 8 connection(s)
I20260924 16:12:10.451719 8887 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:12:10.452860 8887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: Bootstrap starting.
I20260924 16:12:10.464418 8887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:12:10.464790 8887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: Bootstrap complete.
I20260924 16:12:10.465492 8887 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true } }
I20260924 16:12:10.465663 8887 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7b93ed626234c58a2920ac0fe0dd3a7, State: Initialized, Role: LEARNER
I20260924 16:12:10.465806 8887 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true } }
I20260924 16:12:10.466414 8887 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:10.466672 8890 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true } } }
I20260924 16:12:10.466888 8890 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: This master's current role is: LEARNER
I20260924 16:12:10.470218 8901 catalog_manager.cc:1259] Loaded cluster ID: 6c776d6ba04147bdbca859a4c9754ca3
I20260924 16:12:10.470391 8901 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: loading cluster ID for follower catalog manager: success
I20260924 16:12:10.472755 8901 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: acquiring CA information for follower catalog manager: success
I20260924 16:12:10.474399 8901 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 16:12:10.506196 8823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:10.506596 8823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:10.506644 8823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:10.511073 8823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:10.511164 8823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:10.511201 8823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:10.511255 8823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:10.511291 8823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:10.517318 8823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:33025,127.1.253.125:35437,127.1.253.124:40835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.124.keytab
--principal=oryx/127.1.253.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.124:40835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.253.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:10.518589 8823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:10.519703 8823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:10.528625 8909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:10.528649 8907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:10.528838 8823 server_base.cc:1034] running on GCE node
W20260924 16:12:10.528649 8906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:10.529564 8823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:10.530854 8823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:10.532051 8823 hybrid_clock.cc:648] HybridClock initialized: now 1790266330532038 us; error 33 us; skew 500 ppm
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:10.535041 8823 init.cc:377] Logged in from keytab as oryx/127.1.253.124@KRBTEST.COM (short username oryx)
I20260924 16:12:10.536302 8823 webserver.cc:460] Webserver started at http://127.1.253.124:45957/ using document root <none> and password file <none>
I20260924 16:12:10.536875 8823 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:10.536975 8823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:10.537202 8823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:10.539611 8823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data/instance:
uuid: "cda9899a794242d6b9a0dad908ba1553"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-6wvm"
server_key: "d4e9b0206f769e16c30892c3b5eaddf3"
server_key_iv: "7b14933832174db8cae483bf4d86ff36"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.540377 8823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/wal/instance:
uuid: "cda9899a794242d6b9a0dad908ba1553"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-6wvm"
server_key: "d4e9b0206f769e16c30892c3b5eaddf3"
server_key_iv: "7b14933832174db8cae483bf4d86ff36"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.544442 8823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:12:10.547122 8916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.548509 8823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:12:10.548681 8823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/wal
uuid: "cda9899a794242d6b9a0dad908ba1553"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-6wvm"
server_key: "d4e9b0206f769e16c30892c3b5eaddf3"
server_key_iv: "7b14933832174db8cae483bf4d86ff36"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.548806 8823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:10.577708 8823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:10.581107 8823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:10.581318 8823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:10.588347 8823 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.124:40835
I20260924 16:12:10.588637 8968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.124:40835 every 8 connection(s)
I20260924 16:12:10.589357 8823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
I20260924 16:12:10.592469 8969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:10.592592 8442 heartbeater.cc:503] Master 127.1.253.126:33025 was elected leader, sending a full tablet report...
I20260924 16:12:10.596668 8969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:12:10.598549 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 8823
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.124@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.253.124@KRBTEST.COM
I20260924 16:12:10.610736 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.598792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.1.253.126:33025)
0924 16:12:10.598926 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.598930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.599953 (+ 1023us) server_negotiation.cc:407] Connection header received
0924 16:12:10.600854 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.600857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.600921 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.601012 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.601929 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.602523 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.603511 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.603731 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.603841 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.606507 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.606511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.606514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.606544 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.608315 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.608978 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.608981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.608982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.609054 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.609437 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.609440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.609442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.609658 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.609734 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.610431 (+ 697us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.610558 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:12:10.611850 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.599421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.1.253.124:40835)
0924 16:12:10.599845 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.599867 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.599907 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:12:10.600644 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.600663 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.600866 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.601069 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.601908 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.602837 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.603648 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.604076 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.604225 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.607077 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.607082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.607092 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.607112 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.609122 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.609578 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.609583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.609584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.609663 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.609973 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.609976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.609978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.610280 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.610477 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.610690 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.610809 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":326,"thread_start_us":141,"threads_started":1}
I20260924 16:12:10.611850 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.599232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:35816)
0924 16:12:10.599827 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.599867 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.600658 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.601065 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.601071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.601427 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.601790 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.601805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.602664 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.602667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.603387 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.603396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.603878 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.603882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.604083 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.604697 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.604724 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.606342 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.608476 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.608484 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.608496 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.608848 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.609145 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.609152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.609157 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.609333 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.609777 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.609786 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.610126 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.610370 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.610723 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":1,"thread_start_us":136,"threads_started":1}
I20260924 16:12:10.613490 8969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.124@KRBTEST.COM for oryx/127.1.253.125@KRBTEST.COM
I20260924 16:12:10.621446 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.614005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:35437 (local address 127.0.0.1:59536)
0924 16:12:10.614167 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.614183 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.614278 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.614655 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.614658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.614788 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.614880 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.614885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.615820 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.615823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.616416 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.616425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.616899 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.616902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.616976 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.617413 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.617432 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.618716 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.620230 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.620232 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.620234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.620414 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.620663 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.620666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.620667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.620739 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.621067 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.621069 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.621141 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.621232 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.621326 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 16:12:10.621557 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.614090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.1.253.125:35437)
0924 16:12:10.614404 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.614407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.614420 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:12:10.614457 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.614459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.614499 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.614575 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.614986 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.615714 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.616583 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.616805 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.616885 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.618849 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.618852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.618853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.618876 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.620122 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.620499 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.620502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.620504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.620581 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.620813 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.620816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.620817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.620979 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.621065 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.621288 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.621384 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260924 16:12:10.622361 8969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.124@KRBTEST.COM for oryx/127.1.253.124@KRBTEST.COM
I20260924 16:12:10.629576 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.622848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:40835 (local address 127.0.0.1:45794)
0924 16:12:10.622959 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.622974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.623046 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.623281 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.623283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.623408 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.623476 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.623480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.624164 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.624166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.624637 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.624644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.625032 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.625034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.625093 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.625537 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.625548 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.626744 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.628165 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.628169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.628171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.628350 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.628623 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.628626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.628628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.628702 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.629064 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.629067 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.629152 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.629257 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.629383 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 16:12:10.629576 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.622996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.1.253.124:40835)
0924 16:12:10.623104 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.623107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.623118 (+ 11us) server_negotiation.cc:407] Connection header received
0924 16:12:10.623146 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.623148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.623179 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.623248 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.623573 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.624059 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.624729 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.624944 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.625027 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.626860 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.626862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.626864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.626884 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.628054 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.628444 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.628450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.628452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.628526 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.628790 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.628795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.628797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.628951 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.629072 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.629301 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.629431 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":46}
I20260924 16:12:10.632898 8969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: Bootstrap starting.
I20260924 16:12:10.635324 8969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:10.636099 8969 log.cc:824] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:10.638019 8969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: No bootstrap required, opened a new log
I20260924 16:12:10.641387 8969 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } }
I20260924 16:12:10.641607 8969 raft_consensus.cc:374] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:10.641691 8969 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda9899a794242d6b9a0dad908ba1553, State: Initialized, Role: FOLLOWER
I20260924 16:12:10.642139 8969 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } }
I20260924 16:12:10.642650 8975 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } } }
I20260924 16:12:10.642771 8975 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:12:10.643407 8969 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:12:10.648119 8986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:12:10.648219 8986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:12:10.652017 8823 master_runner.cc:428] Detected that this master cda9899a794242d6b9a0dad908ba1553 is joining an existing cluster
I20260924 16:12:10.652078 8823 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.253.124:40835
I20260924 16:12:10.663888 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.655132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.1.253.126:33025)
0924 16:12:10.655382 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.655386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.655400 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:12:10.655478 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.655481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.655532 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.655616 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.656177 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.656994 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.658325 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.658670 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.658738 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.660149 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.660152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.660155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.660204 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.661955 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.662534 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.662541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.662545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.662638 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.662925 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.662930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.662931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.663196 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.663310 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.663592 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.663722 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260924 16:12:10.663923 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.654609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:35832)
0924 16:12:10.654958 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.654973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.655071 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.655655 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.655665 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.655902 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.656063 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.656070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.657114 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.657117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.658162 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.658172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.658914 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.658919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.659070 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.659616 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.659629 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.659998 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.662132 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.662136 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.662138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.662424 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.662734 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.662737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.662739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.662831 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.663300 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.663303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.663404 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.663525 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.663696 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":48,"thread_start_us":119,"threads_started":1}
I20260924 16:12:10.665292 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.655003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.1.253.125:35437)
0924 16:12:10.655273 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.655277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.655327 (+ 50us) server_negotiation.cc:407] Connection header received
0924 16:12:10.655485 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.655488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.655532 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.655616 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.656737 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.657540 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.659771 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.660131 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.660231 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.661488 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.661492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.661495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.661523 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.663332 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.663814 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.663817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.663819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.663899 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.664485 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.664489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.664491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.664679 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.664737 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.665032 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.665153 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260924 16:12:10.665304 8993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.654617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:35437 (local address 127.0.0.1:59550)
0924 16:12:10.655251 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.655263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.655385 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.656206 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.656209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.656468 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.656582 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.656590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.657698 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.657701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.659620 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.659632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.660393 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.660397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.660495 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.660908 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.660922 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.661340 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.663420 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.663422 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.663424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.663704 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.664239 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.664242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.664244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.664373 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.664795 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.664797 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.664864 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.664978 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.665107 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":23,"spinlock_wait_cycles":1792,"thread_start_us":88,"threads_started":1}
I20260924 16:12:10.665819 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 8313
I20260924 16:12:10.676339 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:40341
--local_ip_for_outbound_sockets=127.1.253.65
--tserver_master_addrs=127.1.253.126:33025
--webserver_port=38515
--webserver_interface=127.1.253.65
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab
--principal=oryx/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.253.126:33025,127.1.253.125:35437
--tserver_master_addrs=127.1.253.126:33025,127.1.253.125:35437,127.1.253.124:40835 with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.126@KRBTEST.COM for oryx/127.1.253.124@KRBTEST.COM
I20260924 16:12:10.686579 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.674146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:40835 (local address 127.0.0.1:45806)
0924 16:12:10.674335 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.674355 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.674448 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.676348 (+ 1900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.676351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.676560 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.676691 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.676698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.677890 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.677893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.678817 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.678828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.679456 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.679460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.679551 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.680069 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.680082 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.681413 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.684492 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.684497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.684500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.684804 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.685221 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.685228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.685233 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.685352 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.685912 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.685917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.686043 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.686141 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.686342 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 16:12:10.687505 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.674195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.1.253.124:40835)
0924 16:12:10.676047 (+ 1852us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.676055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.676069 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:12:10.676144 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.676147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.676203 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.676812 (+ 609us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.676923 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.677757 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.678969 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.679328 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.679425 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.681600 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.681604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.681606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.681630 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.684351 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.684973 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.684979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.684984 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.685083 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.685500 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.685507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.685511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.685769 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.687086 (+ 1317us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.687166 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.687300 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1777,"thread_start_us":185,"threads_started":1}
I20260924 16:12:10.687953 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.655119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:40835 (local address 127.0.0.1:45802)
0924 16:12:10.655993 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.656004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.656109 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.656375 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.656378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.656614 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.656718 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.656724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.657649 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.657652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.658508 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.658516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.659100 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.659103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.659191 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.683068 (+ 23877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.683089 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.683940 (+ 851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.686095 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.686098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.686101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.686372 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.686689 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.686693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.686697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.686800 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.687373 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.687376 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.687498 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.687611 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.687756 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260924 16:12:10.688544 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.655787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.1.253.124:40835)
0924 16:12:10.655901 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.655906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.656072 (+ 166us) server_negotiation.cc:407] Connection header received
0924 16:12:10.656196 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.656199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.656232 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.656275 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.656835 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.657535 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.658609 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.658897 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.658964 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.684122 (+ 25158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.684126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.684130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.684161 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.685979 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.686498 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.686501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.686503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.686581 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.686889 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.686892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.686894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.687109 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.687195 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.688256 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.688363 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:12:10.688791 8251 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.253.124:40835: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cda9899a794242d6b9a0dad908ba1553"
member_type: NON_VOTER
last_known_addr {
host: "127.1.253.124"
port: 40835
}
attrs {
promote: true
}
}
dest_uuid: "519c19cb2bb7494d9f685cfa7766ea10"
cas_config_opid_index: 5
I20260924 16:12:10.689455 8251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } }
I20260924 16:12:10.690641 8811 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cda9899a794242d6b9a0dad908ba1553 (127.1.253.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:10.691705 8943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:10.692366 8811 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.253.124:40835
I20260924 16:12:10.692539 8810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:10.692646 8810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:10.693231 8943 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 1 FOLLOWER]: Refusing update from remote peer 519c19cb2bb7494d9f685cfa7766ea10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 16:12:10.693442 8823 master.cc:537] Master@127.1.253.124:40835 shutting down...
I20260924 16:12:10.695372 8810 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:12:10.697530 8236 consensus_peers.cc:487] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 -> Peer cda9899a794242d6b9a0dad908ba1553 (127.1.253.124:40835): Couldn't send request to peer cda9899a794242d6b9a0dad908ba1553. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 16:12:10.699309 8823 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:10.699638 8823 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:10.699750 8823 tablet_replica.cc:330] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: stopping tablet replica
I20260924 16:12:10.701259 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.690581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:35437 (local address 127.0.0.1:59560)
0924 16:12:10.690960 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.690977 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.691060 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.693052 (+ 1992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.693055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.693073 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:10.693163 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.693169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.698357 (+ 5188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.698362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.699624 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.699636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.700833 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.700837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.700934 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.700983 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.701052 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.701129 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300}
I20260924 16:12:10.701697 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.692624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.1.253.125:35437)
0924 16:12:10.692762 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.692767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.692782 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:12:10.692835 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.692837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.692903 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.693014 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:10.693292 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.694137 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.699793 (+ 5656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.700484 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.700566 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.700732 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.701484 (+ 752us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.701569 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:12:10.702178 8861 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 LEARNER]: Refusing update from remote peer 519c19cb2bb7494d9f685cfa7766ea10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 16:12:10.702709 8810 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 16:12:10.704597 8861 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cda9899a794242d6b9a0dad908ba1553 (127.1.253.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:10.708086 8890 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519c19cb2bb7494d9f685cfa7766ea10. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:10.708230 8890 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: This master's current role is: LEARNER
I20260924 16:12:10.708334 8890 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: NON_VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: true } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:10.708422 8890 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: This master's current role is: LEARNER
I20260924 16:12:10.709209 8810 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10: attempting to promote NON_VOTER b7b93ed626234c58a2920ac0fe0dd3a7 to VOTER
I20260924 16:12:10.709723 8810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } }
I20260924 16:12:10.711189 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.710871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:40835 (local address 127.0.0.1:45816)
0924 16:12:10.711017 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.711103 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:40835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 16:12:10.711555 8861 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 LEARNER]: Refusing update from remote peer 519c19cb2bb7494d9f685cfa7766ea10: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 16:12:10.712057 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.711731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:40835 (local address 127.0.0.1:45822)
0924 16:12:10.711916 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.711970 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:40835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260924 16:12:10.712055 8811 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 16:12:10.712246 8236 consensus_peers.cc:487] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 -> Peer cda9899a794242d6b9a0dad908ba1553 (127.1.253.124:40835): Couldn't send request to peer cda9899a794242d6b9a0dad908ba1553. Status: Network error: Client connection negotiation failed: client connection to 127.1.253.124:40835: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 16:12:10.713809 8811 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b7b93ed626234c58a2920ac0fe0dd3a7 (127.1.253.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:10.715471 8810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:10.715619 8810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:10.716009 8810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:10.716120 8810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:10.716446 8861 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b7b93ed626234c58a2920ac0fe0dd3a7 (127.1.253.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:10.718585 8998 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:10.718789 8998 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:12:10.718958 8823 master.cc:552] Master@127.1.253.124:40835 shutdown complete.
I20260924 16:12:10.719013 8823 master_runner.cc:305] Clearing existing system tablet
I20260924 16:12:10.719290 8823 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:10.719336 8823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:10.720522 8823 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 16:12:10.721452 9002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.721670 8823 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:10.721988 8823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:10.722081 8823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/wal
uuid: "cda9899a794242d6b9a0dad908ba1553"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-6wvm"
server_key: "d4e9b0206f769e16c30892c3b5eaddf3"
server_key_iv: "7b14933832174db8cae483bf4d86ff36"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.724319 8823 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:10.727010 8823 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 16:12:10.727207 8823 log.cc:1196] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 16:12:10.727553 8823 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: Deleting consensus metadata
I20260924 16:12:10.727771 8823 master_runner.cc:315] Copying system tablet from 127.1.253.126:33025
I20260924 16:12:10.729032 8823 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: tablet copy: Beginning tablet copy session from remote peer at address 127.1.253.126:33025
I20260924 16:12:10.738989 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.729823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.1.253.126:33025)
0924 16:12:10.730035 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.730041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.730331 (+ 290us) server_negotiation.cc:407] Connection header received
0924 16:12:10.730547 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.730551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.730609 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.730697 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.731257 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.732063 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.733295 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.733628 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.733736 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.735150 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.735155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.735160 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.735188 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.736985 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.737574 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.737579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.737581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.737678 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.738003 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.738006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.738008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.738224 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.738378 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.738698 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.738816 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 16:12:10.739007 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.729705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:35836)
0924 16:12:10.730176 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.730192 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.730389 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.730734 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.730738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.730965 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.731114 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.731126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.732191 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.732195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.733143 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.733155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.733753 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.733757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.733864 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.734516 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.734534 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.734985 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.737106 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.737110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.737113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.737389 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.737781 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.737785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.737787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.737895 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.738374 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.738378 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.738498 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.738623 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.738778 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":94,"threads_started":1}
I20260924 16:12:10.739799 8271 tablet_copy_service.cc:140] P 519c19cb2bb7494d9f685cfa7766ea10: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cda9899a794242d6b9a0dad908ba1553 ({username='oryx', principal='oryx/127.1.253.124@KRBTEST.COM'} at 127.0.0.1:35836)
I20260924 16:12:10.739914 8271 tablet_copy_service.cc:161] P 519c19cb2bb7494d9f685cfa7766ea10: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cda9899a794242d6b9a0dad908ba1553 at {username='oryx', principal='oryx/127.1.253.124@KRBTEST.COM'} at 127.0.0.1:35836: session id = cda9899a794242d6b9a0dad908ba1553-00000000000000000000000000000000
I20260924 16:12:10.741158 8271 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 16:12:10.742591 8823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 16:12:10.745790 8823 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: tablet copy: Starting download of 0 data blocks...
I20260924 16:12:10.746076 8823 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: tablet copy: Starting download of 1 WAL segments...
I20260924 16:12:10.747812 8823 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 16:12:10.749665 8271 tablet_copy_service.cc:342] P 519c19cb2bb7494d9f685cfa7766ea10: Request end of tablet copy session cda9899a794242d6b9a0dad908ba1553-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.253.124@KRBTEST.COM'} at 127.0.0.1:35836
I20260924 16:12:10.749747 8271 tablet_copy_service.cc:433] P 519c19cb2bb7494d9f685cfa7766ea10: ending tablet copy session cda9899a794242d6b9a0dad908ba1553-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cda9899a794242d6b9a0dad908ba1553
I20260924 16:12:10.758546 8823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:10.760581 9009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:10.763352 9010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:10.763799 8823 server_base.cc:1034] running on GCE node
W20260924 16:12:10.766083 9012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:10.766367 8823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:10.766733 8823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:10.767871 8823 hybrid_clock.cc:648] HybridClock initialized: now 1790266330767842 us; error 49 us; skew 500 ppm
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:10.769615 8823 init.cc:377] Logged in from keytab as oryx/127.1.253.124@KRBTEST.COM (short username oryx)
I20260924 16:12:10.770164 8823 webserver.cc:460] Webserver started at http://127.1.253.124:42255/ using document root <none> and password file <none>
I20260924 16:12:10.770464 8823 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:10.770516 8823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:10.772095 8823 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 16:12:10.773267 9019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.773623 8823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 16:12:10.773710 8823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/wal
uuid: "cda9899a794242d6b9a0dad908ba1553"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-6wvm"
server_key: "d4e9b0206f769e16c30892c3b5eaddf3"
server_key_iv: "7b14933832174db8cae483bf4d86ff36"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.773826 8823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:10.784771 8823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:10.787245 8823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:10.795337 8823 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.124:40835
I20260924 16:12:10.796116 8823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
I20260924 16:12:10.798485 9071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.124:40835 every 8 connection(s)
I20260924 16:12:10.800220 9072 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:12:10.801362 9072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: Bootstrap starting.
I20260924 16:12:10.814865 9072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:12:10.815299 9072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: Bootstrap complete.
I20260924 16:12:10.815445 8576 heartbeater.cc:503] Master 127.1.253.126:33025 was elected leader, sending a full tablet report...
I20260924 16:12:10.817168 9072 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } }
I20260924 16:12:10.817332 9072 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cda9899a794242d6b9a0dad908ba1553, State: Initialized, Role: LEARNER
I20260924 16:12:10.817445 9072 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } }
I20260924 16:12:10.818048 9072 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:10.818478 9075 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:10.818738 9075 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: This master's current role is: LEARNER
I20260924 16:12:10.821731 9086 catalog_manager.cc:1259] Loaded cluster ID: 6c776d6ba04147bdbca859a4c9754ca3
I20260924 16:12:10.821846 9086 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: loading cluster ID for follower catalog manager: success
I20260924 16:12:10.824258 9086 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: acquiring CA information for follower catalog manager: success
I20260924 16:12:10.825866 9086 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 16:12:10.834102 8996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:10.834452 8996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:10.834563 8996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:10.838773 8996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:10.838876 8996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:10.838969 8996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:12:10.844633 8996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab
--principal=oryx/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:40341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=38515
--tserver_master_addrs=127.1.253.126:33025,127.1.253.125:35437,127.1.253.124:40835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:10.845896 8996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:10.847087 8996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:10.857280 9094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:10.857321 8996 server_base.cc:1034] running on GCE node
W20260924 16:12:10.857268 9092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:10.857268 9091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:10.857916 8996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:10.858630 8996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:10.859819 8996 hybrid_clock.cc:648] HybridClock initialized: now 1790266330859804 us; error 26 us; skew 500 ppm
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:10.862695 8996 init.cc:377] Logged in from keytab as oryx/127.1.253.65@KRBTEST.COM (short username oryx)
I20260924 16:12:10.863632 8996 webserver.cc:460] Webserver started at http://127.1.253.65:38515/ using document root <none> and password file <none>
I20260924 16:12:10.864288 8996 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:10.864394 8996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:10.869141 8996 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:10.871346 9101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.872726 8996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:10.872875 8996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "e2a33489ba884ef0b2eaf21e7897618b"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "77b3704a110746c8612bf9a914924308"
server_key_iv: "b702c2e7d9605e3ba7932a34c91ab51c"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.873462 8996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:10.889583 8996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:10.893178 8996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:10.893394 8996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:10.894042 8996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:10.894979 8996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:10.895051 8996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.895128 8996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:10.895174 8996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.904639 8996 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:40341
I20260924 16:12:10.904646 9214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:40341 every 8 connection(s)
I20260924 16:12:10.905725 8996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:10.906090 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 8996
I20260924 16:12:10.906347 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 8447
I20260924 16:12:10.916617 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:35059
--local_ip_for_outbound_sockets=127.1.253.66
--tserver_master_addrs=127.1.253.126:33025
--webserver_port=44447
--webserver_interface=127.1.253.66
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab
--principal=oryx/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.253.126:33025,127.1.253.125:35437
--tserver_master_addrs=127.1.253.126:33025,127.1.253.125:35437,127.1.253.124:40835 with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.124@KRBTEST.COM
Sep 24 16:12:10 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.125@KRBTEST.COM
I20260924 16:12:10.926100 9219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.908154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:40599 (local address 127.1.253.124:40835)
0924 16:12:10.909985 (+ 1831us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.910002 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.910906 (+ 904us) server_negotiation.cc:407] Connection header received
0924 16:12:10.911741 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.911745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.911803 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.911861 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.913749 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.914848 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.916645 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.917013 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.917105 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.921443 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.921449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.921452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.921486 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.923392 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.924105 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.924112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.924116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.924208 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.924651 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.924654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.924655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.924927 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.924997 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.925762 (+ 765us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.925900 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1723,"thread_start_us":130,"threads_started":1}
I20260924 16:12:10.926172 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.911400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:44491 (local address 127.1.253.125:35437)
0924 16:12:10.911523 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.911528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.911544 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:12:10.911657 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.911663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.911730 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.911798 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.913750 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.914644 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.916676 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.917057 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.917154 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.921862 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.921867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.921873 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.921910 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.923763 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.924337 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.924343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.924346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.924439 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.924800 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.924807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.924810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.925072 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.925171 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.925774 (+ 603us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.925918 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:12:10.926817 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.910324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:32859 (local address 127.1.253.126:33025)
0924 16:12:10.910479 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.910486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.910505 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:12:10.912418 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.912425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.912495 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.912556 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.913750 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.914612 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.916655 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.917005 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.917104 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.922100 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.922105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.922111 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.922148 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.924581 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.925273 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.925276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.925278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.925360 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.925666 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.925671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.925673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.925887 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.925975 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.926562 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.926678 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:12:10.927626 9220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.908497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.65:32859)
0924 16:12:10.909717 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.909771 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.912219 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.912726 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.912729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.912953 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.913570 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.913593 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.914790 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.914793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.916529 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.916537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.917336 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.917340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.917691 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.918679 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.918704 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.921612 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.924696 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.924702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.924706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.925155 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.925463 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.925466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.925468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.925557 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.926020 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.926023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.926104 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.926263 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.926688 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":413,"spinlock_wait_cycles":10240,"thread_start_us":703,"threads_started":1}
I20260924 16:12:10.927981 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.910989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:35437 (local address 127.1.253.65:44491)
0924 16:12:10.911345 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.911360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.911463 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.912001 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.912013 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.912658 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.913568 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.913594 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.914802 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.914808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.916528 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.916542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.917348 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.917354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.917654 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.918890 (+ 1236us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.918910 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.921673 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.923904 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.923910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.923915 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.924208 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.924574 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.924578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.924579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.924679 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.925258 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.925265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.925450 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.926827 (+ 1377us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.927211 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":229,"spinlock_wait_cycles":18816,"thread_start_us":131,"threads_started":1}
I20260924 16:12:10.928825 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.910393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:40835 (local address 127.1.253.65:40599)
0924 16:12:10.910810 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.910827 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.911438 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.912015 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.912017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.912659 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.913573 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.913593 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.914972 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.914975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.916498 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.916512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.917336 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.917340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.917689 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.918878 (+ 1189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.918895 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.921213 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.923536 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.923542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.923561 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.923975 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.924335 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.924341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.924344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.924531 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.925088 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.925097 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.925451 (+ 354us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.926347 (+ 896us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.928653 (+ 2306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":448,"spinlock_wait_cycles":21248,"thread_start_us":184,"threads_started":1}
I20260924 16:12:10.929111 9217 heartbeater.cc:347] Connected to a master server at 127.1.253.125:35437
I20260924 16:12:10.929383 9217 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:10.929785 9215 heartbeater.cc:347] Connected to a master server at 127.1.253.124:40835
I20260924 16:12:10.929888 9215 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:10.930002 9217 heartbeater.cc:511] Master 127.1.253.125:35437 requested a full tablet report, sending...
I20260924 16:12:10.930104 9215 heartbeater.cc:511] Master 127.1.253.124:40835 requested a full tablet report, sending...
I20260924 16:12:10.930394 9216 heartbeater.cc:347] Connected to a master server at 127.1.253.126:33025
I20260924 16:12:10.930475 9216 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:10.930629 9216 heartbeater.cc:511] Master 127.1.253.126:33025 requested a full tablet report, sending...
I20260924 16:12:10.931669 8251 ts_manager.cc:194] Re-registered known tserver with Master: e2a33489ba884ef0b2eaf21e7897618b (127.1.253.65:40341)
I20260924 16:12:10.931859 9036 ts_manager.cc:194] Registered new tserver with Master: e2a33489ba884ef0b2eaf21e7897618b (127.1.253.65:40341)
I20260924 16:12:10.932442 8251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:32859
I20260924 16:12:10.932912 8851 ts_manager.cc:194] Registered new tserver with Master: e2a33489ba884ef0b2eaf21e7897618b (127.1.253.65:40341)
I20260924 16:12:11.032831 8710 heartbeater.cc:503] Master 127.1.253.126:33025 was elected leader, sending a full tablet report...
W20260924 16:12:11.038906 9223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:11.039331 9223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:11.039443 9223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:11.043576 9223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:11.043659 9223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:11.043753 9223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:12:11.049098 9223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab
--principal=oryx/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:35059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=44447
--tserver_master_addrs=127.1.253.126:33025,127.1.253.125:35437,127.1.253.124:40835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:11.050173 9223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:11.051280 9223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:11.061340 9231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:11.061334 9228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:11.061384 9223 server_base.cc:1034] running on GCE node
W20260924 16:12:11.061334 9229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:11.062045 9223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:11.062743 9223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:11.063908 9223 hybrid_clock.cc:648] HybridClock initialized: now 1790266331063874 us; error 47 us; skew 500 ppm
Sep 24 16:12:11 dist-test-slave-6wvm krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:11.066975 9223 init.cc:377] Logged in from keytab as oryx/127.1.253.66@KRBTEST.COM (short username oryx)
I20260924 16:12:11.067962 9223 webserver.cc:460] Webserver started at http://127.1.253.66:44447/ using document root <none> and password file <none>
I20260924 16:12:11.068499 9223 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:11.068621 9223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:11.072552 9223 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:11.074826 9238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:11.076112 9223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:11.076246 9223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "edc542371ed34c6fa9e7153ad00c242d"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "714d63fdd03532779ff66c39547d5907"
server_key_iv: "ff5e532427dd1bf50051a666a81c7c55"
server_key_version: "encryptionkey@0"
I20260924 16:12:11.076802 9223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:11.095811 9223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:11.099246 9223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:11.099455 9223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:11.100086 9223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:11.100989 9223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:11.101070 9223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:11.101163 9223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:11.101212 9223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:11.110479 9223 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:35059
I20260924 16:12:11.110548 9351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:35059 every 8 connection(s)
I20260924 16:12:11.111727 9223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:11.114570 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 9223
I20260924 16:12:11.114763 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 8581
I20260924 16:12:11.124974 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:35207
--local_ip_for_outbound_sockets=127.1.253.67
--tserver_master_addrs=127.1.253.126:33025
--webserver_port=45517
--webserver_interface=127.1.253.67
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab
--principal=oryx/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.253.126:33025,127.1.253.125:35437
--tserver_master_addrs=127.1.253.126:33025,127.1.253.125:35437,127.1.253.124:40835 with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:12:11 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
Sep 24 16:12:11 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.124@KRBTEST.COM
Sep 24 16:12:11 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.125@KRBTEST.COM
I20260924 16:12:11.132804 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.113940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:38087 (local address 127.1.253.126:33025)
0924 16:12:11.114115 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.114121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.116195 (+ 2074us) server_negotiation.cc:407] Connection header received
0924 16:12:11.117615 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.117619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.117682 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.117812 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.119854 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.120395 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.122529 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.122965 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.123054 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.127552 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.127559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.127563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.127596 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.129527 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.130363 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.130369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.130373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.130475 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.131216 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.131221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.131226 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.131488 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.131586 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.132406 (+ 820us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.132557 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:12:11.134310 9219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.114644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:36083 (local address 127.1.253.124:40835)
0924 16:12:11.114821 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.114827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.123067 (+ 8240us) server_negotiation.cc:407] Connection header received
0924 16:12:11.123229 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.123232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.123300 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.123381 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.124143 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.124966 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.126162 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.126568 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.126636 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.130156 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.130173 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.130176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.130200 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.132361 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.132870 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.132875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.132878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.132964 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.133289 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.133292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.133294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.133527 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.133599 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.133954 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.134093 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 16:12:11.134634 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.115795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:33875 (local address 127.1.253.125:35437)
0924 16:12:11.115909 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.115913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.123474 (+ 7561us) server_negotiation.cc:407] Connection header received
0924 16:12:11.123654 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.123659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.123718 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.123774 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.124402 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.125149 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.126491 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.126803 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.126882 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.131030 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.131036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.131040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.131072 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.132855 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.133302 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.133306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.133307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.133385 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.133701 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.133704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.133706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.133935 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.134012 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.134364 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.134484 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 16:12:11.135456 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.114488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:40835 (local address 127.1.253.66:36083)
0924 16:12:11.122966 (+ 8478us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.122979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.123098 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.123521 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.123524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.123808 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:11.123997 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.124005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.125103 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.125106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.126033 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.126042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.126804 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.126808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.126917 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.127720 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:11.127734 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:11.130015 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:11.132494 (+ 2479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:11.132497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:11.132499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:11.132751 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:11.133077 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:11.133080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:11.133081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:11.133176 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:11.133673 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:11.133679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:11.133771 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.134515 (+ 744us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.134644 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8411,"thread_start_us":140,"threads_started":1}
I20260924 16:12:11.135470 9357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.115625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:35437 (local address 127.1.253.66:33875)
0924 16:12:11.123390 (+ 7765us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.123405 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.123536 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.123922 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.123926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.124109 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:11.124238 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.124244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.125484 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.125487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.126358 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.126366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.127057 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.127060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.127207 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.127710 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:11.127733 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:11.130865 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:11.132963 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:11.132966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:11.132968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:11.133196 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:11.133494 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:11.133497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:11.133499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:11.133585 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:11.134069 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:11.134072 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:11.134153 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.134836 (+ 683us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.134959 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7676,"spinlock_wait_cycles":3072,"thread_start_us":79,"threads_started":1}
I20260924 16:12:11.136716 9352 heartbeater.cc:347] Connected to a master server at 127.1.253.124:40835
I20260924 16:12:11.136766 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.114396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.66:38087)
0924 16:12:11.116019 (+ 1623us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.116078 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.117361 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.118019 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.118031 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.118716 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:11.119676 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.119700 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.120535 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.120542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.122378 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.122392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.123281 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.123287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.123606 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.124467 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:11.124493 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:11.127294 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:11.129706 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:11.129716 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:11.129735 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:11.130206 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:11.130862 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:11.130868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:11.130872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:11.131083 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:11.131667 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:11.131677 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:11.132074 (+ 397us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.133506 (+ 1432us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.133920 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1353,"spinlock_wait_cycles":81792,"thread_start_us":875,"threads_started":1}
I20260924 16:12:11.136806 9354 heartbeater.cc:347] Connected to a master server at 127.1.253.125:35437
I20260924 16:12:11.136981 9352 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:11.137334 9354 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:11.137953 9354 heartbeater.cc:511] Master 127.1.253.125:35437 requested a full tablet report, sending...
I20260924 16:12:11.138063 9353 heartbeater.cc:347] Connected to a master server at 127.1.253.126:33025
I20260924 16:12:11.138129 9353 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:11.138336 9353 heartbeater.cc:511] Master 127.1.253.126:33025 requested a full tablet report, sending...
I20260924 16:12:11.137953 9352 heartbeater.cc:511] Master 127.1.253.124:40835 requested a full tablet report, sending...
I20260924 16:12:11.139235 9036 ts_manager.cc:194] Registered new tserver with Master: edc542371ed34c6fa9e7153ad00c242d (127.1.253.66:35059)
I20260924 16:12:11.139235 8851 ts_manager.cc:194] Registered new tserver with Master: edc542371ed34c6fa9e7153ad00c242d (127.1.253.66:35059)
I20260924 16:12:11.139233 8251 ts_manager.cc:194] Re-registered known tserver with Master: edc542371ed34c6fa9e7153ad00c242d (127.1.253.66:35059)
I20260924 16:12:11.140025 8251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:38087
I20260924 16:12:11.168011 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.163938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:40835 (local address 127.0.0.1:45836)
0924 16:12:11.164076 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.164094 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.164189 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.164513 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.164516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.164529 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:11.164620 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.164624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.165656 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.165666 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.166561 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.166572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.167620 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.167622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.167743 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.167776 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.167830 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.167889 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 16:12:11.168080 9219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.164021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.1.253.124:40835)
0924 16:12:11.164191 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.164197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.164217 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:12:11.164282 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.164286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.164373 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.164492 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:11.164730 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.165537 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.166696 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.167418 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.167607 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.167806 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.167852 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.167936 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:12:11.168967 9046 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 16:12:11.169384 9075 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519c19cb2bb7494d9f685cfa7766ea10. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:11.169538 9075 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: This master's current role is: LEARNER
W20260924 16:12:11.253439 9359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:11.253741 9359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:11.253826 9359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:11.257859 9359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:11.257961 9359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:11.258056 9359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:12:11.263387 9359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab
--principal=oryx/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:35207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=45517
--tserver_master_addrs=127.1.253.126:33025,127.1.253.125:35437,127.1.253.124:40835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:11.264473 9359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:11.265586 9359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:11.276022 9365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:11.276017 9364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:11.276050 9367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:11.276319 9359 server_base.cc:1034] running on GCE node
I20260924 16:12:11.276727 9359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:11.277326 9359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:11.278486 9359 hybrid_clock.cc:648] HybridClock initialized: now 1790266331278459 us; error 31 us; skew 500 ppm
Sep 24 16:12:11 dist-test-slave-6wvm krb5kdc[8206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:11.281216 9359 init.cc:377] Logged in from keytab as oryx/127.1.253.67@KRBTEST.COM (short username oryx)
I20260924 16:12:11.282105 9359 webserver.cc:460] Webserver started at http://127.1.253.67:45517/ using document root <none> and password file <none>
I20260924 16:12:11.282737 9359 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:11.282827 9359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:11.286537 9359 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 16:12:11.288501 9374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:11.289795 9359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:11.289922 9359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "dff39b9cdeb04b469a97c979003e5cf1"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-6wvm"
server_key: "67d27125c9f1b9597a219e76ff7d444c"
server_key_iv: "5705f4365375a31a3c833a65f5115eca"
server_key_version: "encryptionkey@0"
I20260924 16:12:11.290512 9359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:11.299705 9359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:11.302487 9359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:11.302663 9359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:11.303262 9359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:11.304064 9359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:11.304131 9359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:11.304200 9359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:11.304243 9359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:11.313792 9359 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:35207
I20260924 16:12:11.313814 9487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:35207 every 8 connection(s)
I20260924 16:12:11.314884 9359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:11.323951 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 9359
Sep 24 16:12:11 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
Sep 24 16:12:11 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.124@KRBTEST.COM
Sep 24 16:12:11 dist-test-slave-6wvm krb5kdc[8206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.125@KRBTEST.COM
I20260924 16:12:11.333750 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.318331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:35437 (local address 127.1.253.67:52581)
0924 16:12:11.319241 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.319251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.319359 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.319618 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.319621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.320305 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:11.320852 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.320867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.321738 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.321741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.323062 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.323073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.326214 (+ 3141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.326222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.326403 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.327146 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:11.327161 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:11.329004 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:11.331877 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:11.331882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:11.331887 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:11.332211 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:11.332535 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:11.332539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:11.332541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:11.332624 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:11.333030 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:11.333034 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:11.333116 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.333186 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.333300 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":94,"thread_start_us":78,"threads_started":1}
I20260924 16:12:11.333950 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.317182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:40835 (local address 127.1.253.67:51843)
0924 16:12:11.318563 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.318597 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.318932 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.319770 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.319773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.320111 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:11.320855 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.320869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.322000 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.322003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.322865 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.322874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.323442 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.323445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.323606 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.324374 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:11.324391 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:11.327370 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:11.330093 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:11.330100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:11.330116 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:11.330500 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:11.330835 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:11.330838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:11.330841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:11.331050 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:11.331594 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:11.331602 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:11.331920 (+ 318us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.332113 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.332444 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1304,"mutex_wait_us":191,"spinlock_wait_cycles":7296,"thread_start_us":159,"threads_started":1}
I20260924 16:12:11.334414 9219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.316757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:51843 (local address 127.1.253.124:40835)
0924 16:12:11.316896 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.316901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.318677 (+ 1776us) server_negotiation.cc:407] Connection header received
0924 16:12:11.319455 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.319459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.319565 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.319649 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.321060 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.321859 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.323003 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.323302 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.323401 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.328197 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.328200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.328202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.328228 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.329967 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.330609 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.330614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.330619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.330707 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.331193 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.331198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.331199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.331417 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.331537 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.334129 (+ 2592us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.334280 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:12:11.335076 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.318709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:52581 (local address 127.1.253.125:35437)
0924 16:12:11.318853 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.318856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.319334 (+ 478us) server_negotiation.cc:407] Connection header received
0924 16:12:11.319456 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.319459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.319514 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.319602 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.321018 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.321571 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.325744 (+ 4173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.325958 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.326065 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.329135 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.329141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.329146 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.329190 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.331750 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.332331 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.332335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.332337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.332421 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.332728 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.332731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.332733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.332918 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.334728 (+ 1810us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.334805 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.334912 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:12:11.335386 9488 heartbeater.cc:347] Connected to a master server at 127.1.253.124:40835
I20260924 16:12:11.335621 9488 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:11.336196 9488 heartbeater.cc:511] Master 127.1.253.124:40835 requested a full tablet report, sending...
I20260924 16:12:11.337512 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.316822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:41527 (local address 127.1.253.126:33025)
0924 16:12:11.316952 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.316956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.317918 (+ 962us) server_negotiation.cc:407] Connection header received
0924 16:12:11.319133 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.319138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.319193 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.319296 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.321059 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.321854 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.323199 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.323523 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.323612 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.327550 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.327555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.327559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.327586 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.330787 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.335824 (+ 5037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.335828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.335829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.335915 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.336225 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.336230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.336231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.336718 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.336806 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.337212 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.337333 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:12:11.337653 9036 ts_manager.cc:194] Registered new tserver with Master: dff39b9cdeb04b469a97c979003e5cf1 (127.1.253.67:35207)
I20260924 16:12:11.338100 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.317182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.67:41527)
0924 16:12:11.317750 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.317802 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.318916 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.319476 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.319485 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.320111 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:11.320855 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.320869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.321972 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.321975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.323062 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.323073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.323637 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.323640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.323732 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.324473 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:11.324489 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:11.327344 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:11.335425 (+ 8081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:11.335431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:11.335436 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:11.335703 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:11.336031 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:11.336036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:11.336039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:11.336126 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:11.336870 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:11.336875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:11.336977 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.337664 (+ 687us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.337812 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":190,"spinlock_wait_cycles":32768,"thread_start_us":610,"threads_started":1}
I20260924 16:12:11.338493 9490 heartbeater.cc:347] Connected to a master server at 127.1.253.125:35437
I20260924 16:12:11.338583 9490 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:11.338734 9489 heartbeater.cc:347] Connected to a master server at 127.1.253.126:33025
I20260924 16:12:11.338789 9490 heartbeater.cc:511] Master 127.1.253.125:35437 requested a full tablet report, sending...
I20260924 16:12:11.338842 9489 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:11.339076 9489 heartbeater.cc:511] Master 127.1.253.126:33025 requested a full tablet report, sending...
I20260924 16:12:11.339563 8851 ts_manager.cc:194] Registered new tserver with Master: dff39b9cdeb04b469a97c979003e5cf1 (127.1.253.67:35207)
I20260924 16:12:11.339699 8251 ts_manager.cc:194] Re-registered known tserver with Master: dff39b9cdeb04b469a97c979003e5cf1 (127.1.253.67:35207)
I20260924 16:12:11.340342 8251 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:41527
I20260924 16:12:11.340555 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.326492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35838 (local address 127.1.253.126:33025)
0924 16:12:11.328212 (+ 1720us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.328217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.329555 (+ 1338us) server_negotiation.cc:407] Connection header received
0924 16:12:11.329715 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.329720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.329780 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.329880 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.330983 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.331842 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.333044 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.333365 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.333432 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.336106 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.336109 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.336112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.336134 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.337868 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.339117 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.339121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.339123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.339222 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.339570 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.339573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.339575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.339806 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.339923 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.340170 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.340323 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
I20260924 16:12:11.345657 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.326727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.1.253.125:35437)
0924 16:12:11.328383 (+ 1656us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.328387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.329127 (+ 740us) server_negotiation.cc:407] Connection header received
0924 16:12:11.331582 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.331585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.331634 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.331707 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.335155 (+ 3448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.335894 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.336932 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.337275 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.337349 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.342452 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.342456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.342460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.342489 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.344037 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.344453 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.344458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.344459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.344536 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.344796 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.344799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.344801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.344955 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.345103 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.345332 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.345556 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1558,"thread_start_us":98,"threads_started":1}
I20260924 16:12:11.346232 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.327494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.1.253.124:40835)
0924 16:12:11.330218 (+ 2724us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.330222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.330260 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:12:11.330297 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.330299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.330344 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.330428 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.331808 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.332503 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.334616 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.334958 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.335041 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.342576 (+ 7535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.342579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.342582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.342605 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.344287 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.344680 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.344683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.344684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.344756 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.345019 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.345021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.345023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.345242 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.345346 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.345949 (+ 603us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.346064 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2641,"thread_start_us":70,"threads_started":1}
I20260924 16:12:11.343477 8251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:11.347836 8251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:11.350311 8999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: NON_VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: true } } }
I20260924 16:12:11.350427 8999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:11.353497 8999 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10: attempting to promote NON_VOTER cda9899a794242d6b9a0dad908ba1553 to VOTER
I20260924 16:12:11.353819 8999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } }
I20260924 16:12:11.355556 9046 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 1 LEARNER]: Refusing update from remote peer 519c19cb2bb7494d9f685cfa7766ea10: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 16:12:11.355857 8861 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 FOLLOWER]: Refusing update from remote peer 519c19cb2bb7494d9f685cfa7766ea10: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 16:12:11.355955 9506 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 16:12:11.356212 9506 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 16:12:11.357847 8999 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, cda9899a794242d6b9a0dad908ba1553 (127.1.253.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } } }
I20260924 16:12:11.358413 8861 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, cda9899a794242d6b9a0dad908ba1553 (127.1.253.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } } }
I20260924 16:12:11.358551 9046 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, cda9899a794242d6b9a0dad908ba1553 (127.1.253.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } } }
I20260924 16:12:11.359177 8999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } } }
I20260924 16:12:11.359269 8999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:11.359445 8999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } } }
I20260924 16:12:11.359428 9506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } } }
I20260924 16:12:11.359526 8999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:11.359539 9506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:11.360607 9075 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } } }
I20260924 16:12:11.360625 9515 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "519c19cb2bb7494d9f685cfa7766ea10" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } } }
I20260924 16:12:11.360707 9075 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:12:11.360720 9515 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:12:11.368393 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.362937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:35207 (local address 127.0.0.1:60646)
0924 16:12:11.363889 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.363904 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.364079 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.364678 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.364681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.364700 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:11.364801 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.364809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.365741 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.365744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.366717 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.366727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.368005 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.368008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.368091 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.368113 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.368177 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.368244 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":3,"thread_start_us":122,"threads_started":1}
I20260924 16:12:11.368494 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.363209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.1.253.67:35207)
0924 16:12:11.363484 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.363497 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.364003 (+ 506us) server_negotiation.cc:407] Connection header received
0924 16:12:11.364355 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.364364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.364543 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.364690 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:11.364893 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.365621 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.367003 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.367877 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.368000 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.368147 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.368246 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.368345 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":101,"threads_started":1}
I20260924 16:12:11.368690 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.362647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40341 (local address 127.0.0.1:48822)
0924 16:12:11.362787 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.362803 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.362885 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.364594 (+ 1709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.364597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.364614 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:11.364719 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.364729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.366085 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.366099 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.367248 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.367257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.368338 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.368342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.368449 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.368473 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.368518 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.368578 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 16:12:11.369738 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.363201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.1.253.65:40341)
0924 16:12:11.364127 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.364141 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.364164 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:12:11.364247 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.364253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.364454 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.364623 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:11.364847 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.365963 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.367419 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.368204 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.369308 (+ 1104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.369459 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.369510 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.369603 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":122,"threads_started":1}
I20260924 16:12:11.369777 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.363656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:35059 (local address 127.0.0.1:54364)
0924 16:12:11.364839 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.364852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.364936 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.365550 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.365553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.365569 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:11.365657 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.365662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.367135 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.367138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.368352 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.368360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.369407 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.369411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.369506 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.369528 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.369605 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.369666 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"thread_start_us":102,"threads_started":1}
I20260924 16:12:11.369881 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.363821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.1.253.66:35059)
0924 16:12:11.364335 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.364347 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.365022 (+ 675us) server_negotiation.cc:407] Connection header received
0924 16:12:11.365100 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.365105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.365285 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.365423 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:11.365767 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.366944 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.368471 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.369304 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.369402 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.369510 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.369626 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.369703 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":108,"threads_started":1}
I20260924 16:12:11.371604 9422 tablet_service.cc:1467] Processing CreateTablet for tablet 339a27714e6d4654800af9607baada70 (DEFAULT_TABLE table=test-table [id=42115afd0011400bbe853a3ce0e6fc60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:11.372218 9286 tablet_service.cc:1467] Processing CreateTablet for tablet 339a27714e6d4654800af9607baada70 (DEFAULT_TABLE table=test-table [id=42115afd0011400bbe853a3ce0e6fc60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:11.372740 9422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339a27714e6d4654800af9607baada70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:11.372718 9149 tablet_service.cc:1467] Processing CreateTablet for tablet 339a27714e6d4654800af9607baada70 (DEFAULT_TABLE table=test-table [id=42115afd0011400bbe853a3ce0e6fc60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:11.373160 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339a27714e6d4654800af9607baada70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:11.373692 9149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339a27714e6d4654800af9607baada70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:11.382115 9522 tablet_bootstrap.cc:492] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d: Bootstrap starting.
I20260924 16:12:11.382937 9523 tablet_bootstrap.cc:492] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b: Bootstrap starting.
I20260924 16:12:11.381619 9521 tablet_bootstrap.cc:492] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1: Bootstrap starting.
I20260924 16:12:11.384881 9523 tablet_bootstrap.cc:654] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:11.385326 9521 tablet_bootstrap.cc:654] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:11.385658 9523 log.cc:824] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:11.386030 9521 log.cc:824] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:11.387610 9523 tablet_bootstrap.cc:492] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b: No bootstrap required, opened a new log
I20260924 16:12:11.387809 9523 ts_tablet_manager.cc:1397] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 16:12:11.388103 9522 tablet_bootstrap.cc:654] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:11.388235 9521 tablet_bootstrap.cc:492] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1: No bootstrap required, opened a new log
I20260924 16:12:11.388411 9521 ts_tablet_manager.cc:1397] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 16:12:11.389201 9522 log.cc:824] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:11.391820 9522 tablet_bootstrap.cc:492] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d: No bootstrap required, opened a new log
I20260924 16:12:11.392000 9522 ts_tablet_manager.cc:1397] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260924 16:12:11.393225 9521 raft_consensus.cc:348] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } } peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } }
I20260924 16:12:11.393543 9521 raft_consensus.cc:374] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:11.393632 9521 raft_consensus.cc:729] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff39b9cdeb04b469a97c979003e5cf1, State: Initialized, Role: FOLLOWER
I20260924 16:12:11.393589 9523 raft_consensus.cc:348] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } } peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } }
I20260924 16:12:11.393828 9523 raft_consensus.cc:374] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:11.393874 9523 raft_consensus.cc:729] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a33489ba884ef0b2eaf21e7897618b, State: Initialized, Role: FOLLOWER
I20260924 16:12:11.394282 9521 consensus_queue.cc:260] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } } peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } }
I20260924 16:12:11.394282 9523 consensus_queue.cc:260] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } } peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } }
I20260924 16:12:11.395296 9523 ts_tablet_manager.cc:1428] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 16:12:11.395313 9521 ts_tablet_manager.cc:1428] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 16:12:11.396636 9216 heartbeater.cc:503] Master 127.1.253.126:33025 was elected leader, sending a full tablet report...
I20260924 16:12:11.398015 9489 heartbeater.cc:503] Master 127.1.253.126:33025 was elected leader, sending a full tablet report...
I20260924 16:12:11.406759 9522 raft_consensus.cc:348] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } } peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } }
I20260924 16:12:11.407071 9522 raft_consensus.cc:374] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:11.407127 9522 raft_consensus.cc:729] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc542371ed34c6fa9e7153ad00c242d, State: Initialized, Role: FOLLOWER
I20260924 16:12:11.407632 9522 consensus_queue.cc:260] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } } peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } }
W20260924 16:12:11.408473 9218 tablet.cc:2369] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:11.408864 9522 ts_tablet_manager.cc:1428] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d: Time spent starting tablet: real 0.017s user 0.009s sys 0.000s
I20260924 16:12:11.409116 9353 heartbeater.cc:503] Master 127.1.253.126:33025 was elected leader, sending a full tablet report...
I20260924 16:12:11.542847 9528 raft_consensus.cc:482] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:11.543094 9528 raft_consensus.cc:504] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } } peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } }
I20260924 16:12:11.544392 9528 leader_election.cc:288] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff39b9cdeb04b469a97c979003e5cf1 (127.1.253.67:35207), edc542371ed34c6fa9e7153ad00c242d (127.1.253.66:35059)
I20260924 16:12:11.548864 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.544822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:34379 (local address 127.1.253.67:35207)
0924 16:12:11.544944 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.544949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.545041 (+ 92us) server_negotiation.cc:407] Connection header received
0924 16:12:11.545288 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.545291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.545347 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.545401 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:11.545854 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.546687 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.547872 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.548288 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.548456 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.548513 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.548675 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.548745 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:12:11.548879 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.544601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:35207 (local address 127.1.253.65:34379)
0924 16:12:11.544943 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.544971 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.545130 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.545555 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.545557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.545577 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:11.545729 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.545735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.546846 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.546849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.547741 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.547750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.548438 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.548441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.548544 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.548573 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.548659 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.548729 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":82,"thread_start_us":85,"threads_started":1}
I20260924 16:12:11.549604 9442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339a27714e6d4654800af9607baada70" candidate_uuid: "e2a33489ba884ef0b2eaf21e7897618b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff39b9cdeb04b469a97c979003e5cf1" is_pre_election: true
I20260924 16:12:11.549927 9442 raft_consensus.cc:2454] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a33489ba884ef0b2eaf21e7897618b in term 0.
I20260924 16:12:11.550034 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.545198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:37657 (local address 127.1.253.66:35059)
0924 16:12:11.545351 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.545355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.545370 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:12:11.545421 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.545424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.545476 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.545550 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:11.545942 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.546688 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.548558 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.549315 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.549426 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.549479 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.549832 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.549912 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:12:11.550407 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.544693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:35059 (local address 127.1.253.65:37657)
0924 16:12:11.545059 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.545072 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.545132 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.545599 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.545604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.545638 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:11.545806 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.545815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.546802 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.546807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.548393 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.548407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.549582 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.549585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.549683 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.549706 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.550161 (+ 455us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.550271 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":54,"thread_start_us":126,"threads_started":1}
I20260924 16:12:11.550462 9104 leader_election.cc:302] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff39b9cdeb04b469a97c979003e5cf1, e2a33489ba884ef0b2eaf21e7897618b; no voters:
I20260924 16:12:11.550772 9528 raft_consensus.cc:2781] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:11.550892 9528 raft_consensus.cc:482] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:11.550956 9528 raft_consensus.cc:3037] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:11.551101 9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339a27714e6d4654800af9607baada70" candidate_uuid: "e2a33489ba884ef0b2eaf21e7897618b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc542371ed34c6fa9e7153ad00c242d" is_pre_election: true
I20260924 16:12:11.551425 9306 raft_consensus.cc:2454] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a33489ba884ef0b2eaf21e7897618b in term 0.
I20260924 16:12:11.552634 9528 raft_consensus.cc:504] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } } peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } }
I20260924 16:12:11.552928 9528 leader_election.cc:288] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [CANDIDATE]: Term 1 election: Requested vote from peers dff39b9cdeb04b469a97c979003e5cf1 (127.1.253.67:35207), edc542371ed34c6fa9e7153ad00c242d (127.1.253.66:35059)
I20260924 16:12:11.553337 9442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339a27714e6d4654800af9607baada70" candidate_uuid: "e2a33489ba884ef0b2eaf21e7897618b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff39b9cdeb04b469a97c979003e5cf1"
I20260924 16:12:11.553517 9442 raft_consensus.cc:3037] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:11.554430 9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339a27714e6d4654800af9607baada70" candidate_uuid: "e2a33489ba884ef0b2eaf21e7897618b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc542371ed34c6fa9e7153ad00c242d"
I20260924 16:12:11.554562 9306 raft_consensus.cc:3037] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:11.555195 9442 raft_consensus.cc:2454] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a33489ba884ef0b2eaf21e7897618b in term 1.
I20260924 16:12:11.555543 9104 leader_election.cc:302] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff39b9cdeb04b469a97c979003e5cf1, e2a33489ba884ef0b2eaf21e7897618b; no voters:
I20260924 16:12:11.555725 9528 raft_consensus.cc:2781] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:11.555850 9306 raft_consensus.cc:2454] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a33489ba884ef0b2eaf21e7897618b in term 1.
I20260924 16:12:11.555912 9528 raft_consensus.cc:686] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 1 LEADER]: Becoming Leader. State: Replica: e2a33489ba884ef0b2eaf21e7897618b, State: Running, Role: LEADER
I20260924 16:12:11.556324 9528 consensus_queue.cc:237] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } } peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } }
I20260924 16:12:11.561132 8251 catalog_manager.cc:5495] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b reported cstate change: term changed from 0 to 1, leader changed from <none> to e2a33489ba884ef0b2eaf21e7897618b (127.1.253.65). New cstate: current_term: 1 leader_uuid: "e2a33489ba884ef0b2eaf21e7897618b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } health_report { overall_health: UNKNOWN } } }
W20260924 16:12:11.566288 9491 tablet.cc:2369] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:11.612095 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.608241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.1.253.65:40341)
0924 16:12:11.608404 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.608411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.608427 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:12:11.608519 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.608521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.608580 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.608700 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:11.608945 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.609462 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.610438 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.610729 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.610841 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.611187 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:11.611702 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:11.611832 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.611870 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.611931 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 16:12:11.613593 9355 tablet.cc:2369] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:11.618862 9442 raft_consensus.cc:1261] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1 [term 1 FOLLOWER]: Refusing update from remote peer e2a33489ba884ef0b2eaf21e7897618b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:11.619725 9528 consensus_queue.cc:1035] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:11.619907 9306 raft_consensus.cc:1261] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d [term 1 FOLLOWER]: Refusing update from remote peer e2a33489ba884ef0b2eaf21e7897618b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:11.620594 9528 consensus_queue.cc:1035] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:11.640987 9537 mvcc.cc:205] Tried to move back new op lower bound from 7332930894301851648 to 7332930894058082304. Current Snapshot: MvccSnapshot[applied={T|T < 7332930894301851648}]
I20260924 16:12:11.643625 8251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:11.643798 8251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:11.648921 8251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 519c19cb2bb7494d9f685cfa7766ea10: Sending DeleteTablet for 3 replicas of tablet 339a27714e6d4654800af9607baada70
I20260924 16:12:11.649601 9149 tablet_service.cc:1514] Processing DeleteTablet for tablet 339a27714e6d4654800af9607baada70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:11 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:48822
I20260924 16:12:11.650432 9286 tablet_service.cc:1514] Processing DeleteTablet for tablet 339a27714e6d4654800af9607baada70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:11 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:54364
I20260924 16:12:11.650449 9422 tablet_service.cc:1514] Processing DeleteTablet for tablet 339a27714e6d4654800af9607baada70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:11 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:60646
I20260924 16:12:11.651144 9545 tablet_replica.cc:330] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d: stopping tablet replica
I20260924 16:12:11.651508 9545 raft_consensus.cc:2229] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:11.651911 9545 raft_consensus.cc:2258] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:11.652587 9544 tablet_replica.cc:330] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b: stopping tablet replica
I20260924 16:12:11.652915 9544 raft_consensus.cc:2229] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:11.653261 9544 raft_consensus.cc:2258] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:11.654237 9545 ts_tablet_manager.cc:1905] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:11.654228 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 8219
I20260924 16:12:11.660820 9545 ts_tablet_manager.cc:1918] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:11.660938 9545 log.cc:1196] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/339a27714e6d4654800af9607baada70
I20260924 16:12:11.661229 9546 tablet_replica.cc:330] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1: stopping tablet replica
I20260924 16:12:11.661284 9545 ts_tablet_manager.cc:1939] T 339a27714e6d4654800af9607baada70 P edc542371ed34c6fa9e7153ad00c242d: Deleting consensus metadata
I20260924 16:12:11.661700 9546 raft_consensus.cc:2229] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:11.662078 9546 raft_consensus.cc:2258] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:11.663813 9546 ts_tablet_manager.cc:1905] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 16:12:11.665004 9239 connection.cc:535] server connection from 127.0.0.1:54364 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 16:12:11.668138 9546 ts_tablet_manager.cc:1918] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:11.670085 9544 ts_tablet_manager.cc:1905] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:11.670734 9546 log.cc:1196] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/339a27714e6d4654800af9607baada70
I20260924 16:12:11.671154 9546 ts_tablet_manager.cc:1939] T 339a27714e6d4654800af9607baada70 P dff39b9cdeb04b469a97c979003e5cf1: Deleting consensus metadata
I20260924 16:12:11.671690 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.652618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.1.253.125:35437)
0924 16:12:11.652741 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.652746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.656083 (+ 3337us) server_negotiation.cc:407] Connection header received
0924 16:12:11.656222 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.656226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.656283 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.656359 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.662626 (+ 6267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.663426 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.664806 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.665240 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.665316 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.667853 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.667876 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.667879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.667909 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.669741 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.670189 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.670192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.670194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.670292 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.670620 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.670625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.670628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.670861 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.670987 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.671397 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.671542 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 16:12:11.672696 9377 connection.cc:441] server connection from 127.0.0.1:60646 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60646 (request call id 1) could send its response
I20260924 16:12:11.672792 9544 ts_tablet_manager.cc:1918] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:11.672876 9544 log.cc:1196] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-0/wal/wals/339a27714e6d4654800af9607baada70
I20260924 16:12:11.673188 9544 ts_tablet_manager.cc:1939] T 339a27714e6d4654800af9607baada70 P e2a33489ba884ef0b2eaf21e7897618b: Deleting consensus metadata
W20260924 16:12:11.676012 9105 connection.cc:441] server connection from 127.0.0.1:48822 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48822 (request call id 1) could send its response
I20260924 16:12:11.683882 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.674440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.1.253.125:35437)
0924 16:12:11.674591 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.674595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.674755 (+ 160us) server_negotiation.cc:407] Connection header received
0924 16:12:11.674908 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.674911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.674963 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.675026 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.675990 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.676716 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.677955 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.678445 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.678533 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.680360 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.680366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.680373 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.680404 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.682096 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.682590 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.682593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.682595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.682680 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.683000 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.683003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.683005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.683212 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.683312 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.683670 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.683770 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:12:11.684181 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.652831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.1.253.124:40835)
0924 16:12:11.653193 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.653198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.673195 (+ 19997us) server_negotiation.cc:407] Connection header received
0924 16:12:11.673463 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.673469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.673529 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.673638 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.674702 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.675537 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.676757 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.677133 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.677209 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.678889 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.678893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.678896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.678926 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.681603 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.682220 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.682223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.682225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.682440 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.682794 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.682797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.682799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.683036 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.683847 (+ 811us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.683913 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.684026 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260924 16:12:11.685951 9219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.674948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45858 (local address 127.1.253.124:40835)
0924 16:12:11.675095 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.675099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.676470 (+ 1371us) server_negotiation.cc:407] Connection header received
0924 16:12:11.676635 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.676638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.676681 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.676728 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.677753 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.678660 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.679865 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.680252 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.680329 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.682023 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.682028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.682030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.682055 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.683733 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.684742 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.684746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.684748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.684832 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.685123 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.685126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.685128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.685311 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.685405 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.685729 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.685815 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:12:12.635804 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:12.635282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.66:35497)
0924 16:12:12.635570 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:12.635689 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":102,"threads_started":1}
W20260924 16:12:12.636399 9353 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:33025 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
I20260924 16:12:12.636477 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:12.635683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.67:47635)
0924 16:12:12.636005 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:12.636322 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":67,"threads_started":1}
W20260924 16:12:12.636802 9489 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:33025 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
I20260924 16:12:12.637609 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:12.637080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.65:53529)
0924 16:12:12.637425 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:12.637526 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":84,"threads_started":1}
W20260924 16:12:12.637880 9216 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:33025 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
I20260924 16:12:13.149682 9557 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 519c19cb2bb7494d9f685cfa7766ea10)
I20260924 16:12:13.149920 9557 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } }
I20260924 16:12:13.151094 9557 leader_election.cc:288] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 519c19cb2bb7494d9f685cfa7766ea10 (127.1.253.126:33025), cda9899a794242d6b9a0dad908ba1553 (127.1.253.124:40835)
I20260924 16:12:13.152091 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.151339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:36332)
0924 16:12:13.151826 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.151932 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":71,"threads_started":1}
W20260924 16:12:13.152431 8837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
I20260924 16:12:13.153043 9561 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 519c19cb2bb7494d9f685cfa7766ea10)
I20260924 16:12:13.153240 9561 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } }
I20260924 16:12:13.153703 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.153377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:36352)
0924 16:12:13.153549 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.153608 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260924 16:12:13.153828 8837 leader_election.cc:334] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 519c19cb2bb7494d9f685cfa7766ea10 (127.1.253.126:33025): Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
I20260924 16:12:13.154362 9561 leader_election.cc:288] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 519c19cb2bb7494d9f685cfa7766ea10 (127.1.253.126:33025), b7b93ed626234c58a2920ac0fe0dd3a7 (127.1.253.125:35437)
I20260924 16:12:13.155005 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.154479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:36368)
0924 16:12:13.154831 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.154908 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":61,"threads_started":1}
W20260924 16:12:13.155210 9022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
I20260924 16:12:13.156028 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.155820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:36376)
0924 16:12:13.155924 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.155969 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 16:12:13.156198 9022 leader_election.cc:334] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 519c19cb2bb7494d9f685cfa7766ea10 (127.1.253.126:33025): Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
I20260924 16:12:13.158319 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.151947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.1.253.124:40835)
0924 16:12:13.153043 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:13.153052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:13.153073 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:12:13.153878 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:13.153882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:13.153988 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:13.154119 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:13.154529 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.155410 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.157287 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.157749 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.157868 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.157932 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:13.158096 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 16:12:13.158169 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":69,"threads_started":1}
I20260924 16:12:13.158320 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.151442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:40835 (local address 127.0.0.1:45862)
0924 16:12:13.152742 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.152756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:13.152851 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:13.154162 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:13.154165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:13.154186 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:13.154375 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.154383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.155575 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.155581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.157134 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.157150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.157869 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.157872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.157962 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.157984 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:12:13.158081 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:12:13.158158 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"thread_start_us":62,"threads_started":1}
I20260924 16:12:13.159026 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cda9899a794242d6b9a0dad908ba1553" is_pre_election: true
I20260924 16:12:13.159245 9046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7b93ed626234c58a2920ac0fe0dd3a7 in term 1.
I20260924 16:12:13.159603 8836 leader_election.cc:302] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b7b93ed626234c58a2920ac0fe0dd3a7, cda9899a794242d6b9a0dad908ba1553; no voters: 519c19cb2bb7494d9f685cfa7766ea10
I20260924 16:12:13.160017 9557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 16:12:13.160082 9557 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 519c19cb2bb7494d9f685cfa7766ea10)
I20260924 16:12:13.160130 9557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:12:13.160507 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.154528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:35437 (local address 127.0.0.1:59596)
0924 16:12:13.154888 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.154926 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:13.155007 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:13.156440 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:13.156443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:13.156459 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:13.156568 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.156574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.157392 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.157399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.158906 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.158922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.159825 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.159832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.159983 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.160028 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 16:12:13.160204 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 16:12:13.160311 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":57,"threads_started":1}
I20260924 16:12:13.161114 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.154624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.1.253.125:35437)
0924 16:12:13.154908 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:13.156201 (+ 1293us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:13.156221 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:12:13.156266 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:13.156268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:13.156310 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:13.156425 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:13.156673 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.157197 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.159060 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.159633 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.159739 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.159796 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:13.160811 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0924 16:12:13.160917 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
I20260924 16:12:13.161345 9557 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } }
I20260924 16:12:13.161546 8861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cda9899a794242d6b9a0dad908ba1553" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" is_pre_election: true
I20260924 16:12:13.161731 8861 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cda9899a794242d6b9a0dad908ba1553 in current term 2: Already voted for candidate b7b93ed626234c58a2920ac0fe0dd3a7 in this term.
I20260924 16:12:13.161792 9557 leader_election.cc:288] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [CANDIDATE]: Term 2 election: Requested vote from peers 519c19cb2bb7494d9f685cfa7766ea10 (127.1.253.126:33025), cda9899a794242d6b9a0dad908ba1553 (127.1.253.124:40835)
I20260924 16:12:13.162204 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.161928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:36382)
0924 16:12:13.162077 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.162133 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 16:12:13.162289 9021 leader_election.cc:302] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cda9899a794242d6b9a0dad908ba1553; no voters: 519c19cb2bb7494d9f685cfa7766ea10, b7b93ed626234c58a2920ac0fe0dd3a7
I20260924 16:12:13.162689 9561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:12:13.163005 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.162764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:36386)
0924 16:12:13.162888 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.162943 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 16:12:13.163187 8837 leader_election.cc:334] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 519c19cb2bb7494d9f685cfa7766ea10 (127.1.253.126:33025): Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
I20260924 16:12:13.163853 9561 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:12:13.164064 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cda9899a794242d6b9a0dad908ba1553"
I20260924 16:12:13.164988 9046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7b93ed626234c58a2920ac0fe0dd3a7 in term 2.
I20260924 16:12:13.165314 8836 leader_election.cc:302] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b7b93ed626234c58a2920ac0fe0dd3a7, cda9899a794242d6b9a0dad908ba1553; no voters: 519c19cb2bb7494d9f685cfa7766ea10
I20260924 16:12:13.165609 9557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:12:13.165880 9557 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [term 2 LEADER]: Becoming Leader. State: Replica: b7b93ed626234c58a2920ac0fe0dd3a7, State: Running, Role: LEADER
I20260924 16:12:13.166276 9557 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } }
I20260924 16:12:13.167779 9567 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7b93ed626234c58a2920ac0fe0dd3a7. Latest consensus state: current_term: 2 leader_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } } }
I20260924 16:12:13.167905 9567 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:13.168280 9569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:13.171784 9569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:13.172458 9569 catalog_manager.cc:1259] Loaded cluster ID: 6c776d6ba04147bdbca859a4c9754ca3
I20260924 16:12:13.172528 9569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:13.173519 9569 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:13.174191 9569 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: Loaded TSK: 0
I20260924 16:12:13.175233 9569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:12:13.214520 8851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:13.217643 8851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:13.221030 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.220758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:36394)
0924 16:12:13.220881 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.220963 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 16:12:13.221156 9046 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [term 2 FOLLOWER]: Refusing update from remote peer b7b93ed626234c58a2920ac0fe0dd3a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 16:12:13.221766 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.221554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.0.0.1:36402)
0924 16:12:13.221659 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.221706 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 16:12:13.221693 9557 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 16:12:13.221880 8837 consensus_peers.cc:487] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 -> Peer 519c19cb2bb7494d9f685cfa7766ea10 (127.1.253.126:33025): Couldn't send request to peer 519c19cb2bb7494d9f685cfa7766ea10. Status: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 16:12:13.223279 9561 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7b93ed626234c58a2920ac0fe0dd3a7. Latest consensus state: current_term: 2 leader_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } } }
I20260924 16:12:13.223397 9561 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:12:13.224694 9557 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } } }
I20260924 16:12:13.224808 9557 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:13.224977 9561 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "519c19cb2bb7494d9f685cfa7766ea10" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33025 } } peers { permanent_uuid: "b7b93ed626234c58a2920ac0fe0dd3a7" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 35437 } attrs { promote: false } } peers { permanent_uuid: "cda9899a794242d6b9a0dad908ba1553" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 40835 } attrs { promote: false } } }
I20260924 16:12:13.225062 9571 mvcc.cc:205] Tried to move back new op lower bound from 7332930900868116480 to 7332930900652584960. Current Snapshot: MvccSnapshot[applied={T|T < 7332930900868116480}]
I20260924 16:12:13.225119 9561 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda9899a794242d6b9a0dad908ba1553 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:12:13.235553 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.230996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:35059 (local address 127.0.0.1:54370)
0924 16:12:13.231249 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.231264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:13.231349 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:13.232536 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:13.232539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:13.232556 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:13.232634 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.232639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.233388 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.233391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.234332 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.234341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.235203 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.235206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.235272 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.235294 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:12:13.235354 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 16:12:13.235409 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260924 16:12:13.235642 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.231525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.1.253.66:35059)
0924 16:12:13.232310 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:13.232314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:13.232334 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:12:13.232388 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:13.232389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:13.232427 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:13.232527 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:13.232748 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.233291 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.234461 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.235071 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.235173 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.235237 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:13.235393 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 16:12:13.235481 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":85,"threads_started":1}
I20260924 16:12:13.236303 9286 tablet_service.cc:1467] Processing CreateTablet for tablet a968e32182ef41d9aa10f5728aadb07c (DEFAULT_TABLE table=test-table [id=23941e5d14b24776897ba03f9a2970c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:13.236624 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a968e32182ef41d9aa10f5728aadb07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:13.236759 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.230710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:35207 (local address 127.0.0.1:60654)
0924 16:12:13.231101 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.231119 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:13.231204 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:13.232098 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:13.232101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:13.232119 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:13.232231 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.232236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.233273 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.233276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.234108 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.234120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.236333 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.236337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.236424 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.236445 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:12:13.236557 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:12:13.236633 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":3}
I20260924 16:12:13.236770 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.230833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.1.253.67:35207)
0924 16:12:13.231837 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:13.231844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:13.231861 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:12:13.231919 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:13.231922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:13.231971 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:13.232072 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:13.232388 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.233153 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.235579 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.236211 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.236334 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.236397 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:13.236585 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 16:12:13.236655 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":920,"thread_start_us":117,"threads_started":1}
I20260924 16:12:13.237350 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.231445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40341 (local address 127.0.0.1:48842)
0924 16:12:13.231716 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.231731 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:13.231816 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:13.232071 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:13.232074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:13.232090 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:13.232199 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.232206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.234281 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.234288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.235427 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.235436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.236926 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.236933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.237042 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.237067 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:12:13.237161 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 16:12:13.237232 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":101,"threads_started":1}
I20260924 16:12:13.237358 9422 tablet_service.cc:1467] Processing CreateTablet for tablet a968e32182ef41d9aa10f5728aadb07c (DEFAULT_TABLE table=test-table [id=23941e5d14b24776897ba03f9a2970c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:13.237613 9422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a968e32182ef41d9aa10f5728aadb07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:13.238220 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.231565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.1.253.65:40341)
0924 16:12:13.231831 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:13.231836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:13.231854 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:12:13.231908 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:13.231911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:13.231962 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:13.232033 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:13.232331 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.233153 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.235565 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.236449 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.236565 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.236635 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:13.237181 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 16:12:13.238090 (+ 909us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260924 16:12:13.238711 9149 tablet_service.cc:1467] Processing CreateTablet for tablet a968e32182ef41d9aa10f5728aadb07c (DEFAULT_TABLE table=test-table [id=23941e5d14b24776897ba03f9a2970c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:13.238992 9149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a968e32182ef41d9aa10f5728aadb07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:13.239426 9579 tablet_bootstrap.cc:492] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d: Bootstrap starting.
I20260924 16:12:13.240113 9580 tablet_bootstrap.cc:492] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1: Bootstrap starting.
I20260924 16:12:13.240442 9579 tablet_bootstrap.cc:654] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:13.241250 9580 tablet_bootstrap.cc:654] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:13.241570 9582 tablet_bootstrap.cc:492] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b: Bootstrap starting.
I20260924 16:12:13.241722 9579 tablet_bootstrap.cc:492] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d: No bootstrap required, opened a new log
I20260924 16:12:13.241819 9579 ts_tablet_manager.cc:1397] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:13.242343 9579 raft_consensus.cc:348] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } } peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } }
I20260924 16:12:13.242487 9579 raft_consensus.cc:374] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:13.242538 9579 raft_consensus.cc:729] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc542371ed34c6fa9e7153ad00c242d, State: Initialized, Role: FOLLOWER
I20260924 16:12:13.242547 9582 tablet_bootstrap.cc:654] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:13.242700 9579 consensus_queue.cc:260] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } } peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } }
I20260924 16:12:13.243113 9579 ts_tablet_manager.cc:1428] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 16:12:13.243183 9580 tablet_bootstrap.cc:492] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1: No bootstrap required, opened a new log
I20260924 16:12:13.243280 9580 ts_tablet_manager.cc:1397] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:13.243646 9582 tablet_bootstrap.cc:492] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b: No bootstrap required, opened a new log
I20260924 16:12:13.243726 9582 ts_tablet_manager.cc:1397] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:12:13.243750 9580 raft_consensus.cc:348] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } } peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } }
I20260924 16:12:13.243868 9580 raft_consensus.cc:374] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:13.243908 9580 raft_consensus.cc:729] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff39b9cdeb04b469a97c979003e5cf1, State: Initialized, Role: FOLLOWER
I20260924 16:12:13.244033 9580 consensus_queue.cc:260] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } } peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } }
I20260924 16:12:13.244097 9582 raft_consensus.cc:348] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } } peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } }
I20260924 16:12:13.244202 9582 raft_consensus.cc:374] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:13.244237 9582 raft_consensus.cc:729] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a33489ba884ef0b2eaf21e7897618b, State: Initialized, Role: FOLLOWER
I20260924 16:12:13.244380 9580 ts_tablet_manager.cc:1428] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:13.244365 9582 consensus_queue.cc:260] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } } peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } }
I20260924 16:12:13.244688 9582 ts_tablet_manager.cc:1428] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:13.244720 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.243582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.66:58977)
0924 16:12:13.244541 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.244605 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":105,"threads_started":1}
I20260924 16:12:13.246151 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.245089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.65:51227)
0924 16:12:13.245980 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.246041 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":104,"threads_started":1}
I20260924 16:12:13.246948 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.245089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.67:57591)
0924 16:12:13.246679 (+ 1590us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.246744 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1239,"thread_start_us":107,"threads_started":1}
I20260924 16:12:13.248013 9354 heartbeater.cc:503] Master 127.1.253.125:35437 was elected leader, sending a full tablet report...
I20260924 16:12:13.248319 9217 heartbeater.cc:503] Master 127.1.253.125:35437 was elected leader, sending a full tablet report...
I20260924 16:12:13.248358 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.248123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.66:35797)
0924 16:12:13.248226 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.248288 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 16:12:13.248634 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.248422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.65:57941)
0924 16:12:13.248519 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.248571 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 16:12:13.248908 9490 heartbeater.cc:503] Master 127.1.253.125:35437 was elected leader, sending a full tablet report...
I20260924 16:12:13.249322 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.249150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.67:46083)
0924 16:12:13.249235 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.249277 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 16:12:13.267944 9587 raft_consensus.cc:482] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:13.268113 9587 raft_consensus.cc:504] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } } peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } }
I20260924 16:12:13.269415 9587 leader_election.cc:288] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edc542371ed34c6fa9e7153ad00c242d (127.1.253.66:35059), e2a33489ba884ef0b2eaf21e7897618b (127.1.253.65:40341)
I20260924 16:12:13.274533 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.269668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:56299 (local address 127.1.253.66:35059)
0924 16:12:13.269795 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:13.269801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:13.269885 (+ 84us) server_negotiation.cc:407] Connection header received
0924 16:12:13.270039 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:13.270042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:13.270115 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:13.270196 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:13.270520 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.271335 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.273284 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.273898 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.274020 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.274071 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:13.274339 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 16:12:13.274416 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:12:13.274533 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.269603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:35059 (local address 127.1.253.67:56299)
0924 16:12:13.269794 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.269809 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:13.269921 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:13.270224 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:13.270229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:13.270278 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:13.270403 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.270409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.271473 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.271480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.273153 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.273165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.274032 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.274035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.274115 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.274165 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 16:12:13.274316 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 16:12:13.274396 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":132}
I20260924 16:12:13.274691 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.269785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:38577 (local address 127.1.253.65:40341)
0924 16:12:13.269912 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:13.269916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:13.270125 (+ 209us) server_negotiation.cc:407] Connection header received
0924 16:12:13.270256 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:13.270259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:13.270312 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:13.270388 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:13.270648 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.271379 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.273284 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.273897 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.274020 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.274071 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:13.274469 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 16:12:13.274542 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:12:13.274708 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.269630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40341 (local address 127.1.253.67:38577)
0924 16:12:13.270032 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.270042 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:13.270130 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:13.270408 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:13.270410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:13.270425 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:13.270545 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.270550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.271509 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.271512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.273154 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:13.273165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.274157 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.274161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:13.274329 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.274353 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:12:13.274451 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 16:12:13.274525 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":202,"thread_start_us":119,"threads_started":1}
I20260924 16:12:13.275214 9169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a968e32182ef41d9aa10f5728aadb07c" candidate_uuid: "dff39b9cdeb04b469a97c979003e5cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a33489ba884ef0b2eaf21e7897618b" is_pre_election: true
I20260924 16:12:13.275266 9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a968e32182ef41d9aa10f5728aadb07c" candidate_uuid: "dff39b9cdeb04b469a97c979003e5cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc542371ed34c6fa9e7153ad00c242d" is_pre_election: true
I20260924 16:12:13.275405 9306 raft_consensus.cc:2454] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff39b9cdeb04b469a97c979003e5cf1 in term 0.
I20260924 16:12:13.275465 9169 raft_consensus.cc:2454] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff39b9cdeb04b469a97c979003e5cf1 in term 0.
I20260924 16:12:13.275775 9375 leader_election.cc:302] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff39b9cdeb04b469a97c979003e5cf1, edc542371ed34c6fa9e7153ad00c242d; no voters:
I20260924 16:12:13.276093 9587 raft_consensus.cc:2781] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:13.276178 9587 raft_consensus.cc:482] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:13.276233 9587 raft_consensus.cc:3037] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:13.277360 9587 raft_consensus.cc:504] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } } peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } }
I20260924 16:12:13.277711 9587 leader_election.cc:288] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [CANDIDATE]: Term 1 election: Requested vote from peers edc542371ed34c6fa9e7153ad00c242d (127.1.253.66:35059), e2a33489ba884ef0b2eaf21e7897618b (127.1.253.65:40341)
I20260924 16:12:13.278066 9169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a968e32182ef41d9aa10f5728aadb07c" candidate_uuid: "dff39b9cdeb04b469a97c979003e5cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a33489ba884ef0b2eaf21e7897618b"
I20260924 16:12:13.278069 9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a968e32182ef41d9aa10f5728aadb07c" candidate_uuid: "dff39b9cdeb04b469a97c979003e5cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc542371ed34c6fa9e7153ad00c242d"
I20260924 16:12:13.278182 9169 raft_consensus.cc:3037] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:13.278198 9306 raft_consensus.cc:3037] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:13.279305 9169 raft_consensus.cc:2454] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff39b9cdeb04b469a97c979003e5cf1 in term 1.
I20260924 16:12:13.279305 9306 raft_consensus.cc:2454] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff39b9cdeb04b469a97c979003e5cf1 in term 1.
I20260924 16:12:13.279596 9375 leader_election.cc:302] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: dff39b9cdeb04b469a97c979003e5cf1, e2a33489ba884ef0b2eaf21e7897618b, edc542371ed34c6fa9e7153ad00c242d; no voters:
I20260924 16:12:13.279805 9587 raft_consensus.cc:2781] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:13.280043 9587 raft_consensus.cc:686] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 1 LEADER]: Becoming Leader. State: Replica: dff39b9cdeb04b469a97c979003e5cf1, State: Running, Role: LEADER
I20260924 16:12:13.280319 9587 consensus_queue.cc:237] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } } peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } }
I20260924 16:12:13.280803 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.280557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.67:46531)
0924 16:12:13.280676 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.280733 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 16:12:13.282692 8851 catalog_manager.cc:5495] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff39b9cdeb04b469a97c979003e5cf1 (127.1.253.67). New cstate: current_term: 1 leader_uuid: "dff39b9cdeb04b469a97c979003e5cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff39b9cdeb04b469a97c979003e5cf1" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 35207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 } health_report { overall_health: UNKNOWN } } }
I20260924 16:12:13.300053 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.296426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.1.253.67:35207)
0924 16:12:13.296571 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:13.296575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:13.296590 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:12:13.296712 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:13.296715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:13.296773 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:13.296886 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:13.297144 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.297648 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.298581 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.298799 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.298907 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.299204 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:13.299659 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:13.299788 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:13.299848 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 16:12:13.299903 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:12:13.302974 9306 raft_consensus.cc:1261] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d [term 1 FOLLOWER]: Refusing update from remote peer dff39b9cdeb04b469a97c979003e5cf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:13.303412 9169 raft_consensus.cc:1261] T a968e32182ef41d9aa10f5728aadb07c P e2a33489ba884ef0b2eaf21e7897618b [term 1 FOLLOWER]: Refusing update from remote peer dff39b9cdeb04b469a97c979003e5cf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:13.303540 9592 consensus_queue.cc:1035] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc542371ed34c6fa9e7153ad00c242d" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:13.304015 9592 consensus_queue.cc:1035] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a33489ba884ef0b2eaf21e7897618b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:13.305893 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.305585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.66:44443)
0924 16:12:13.305755 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.305814 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 16:12:13.307277 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.306779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.65:53229)
0924 16:12:13.307137 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.307195 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
I20260924 16:12:13.309513 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.309121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.67:40855)
0924 16:12:13.309387 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.309440 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260924 16:12:13.310300 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.309991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.67:58097)
0924 16:12:13.310152 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.310214 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 16:12:13.314148 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.313731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.65:49581)
0924 16:12:13.314042 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.314094 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
I20260924 16:12:13.317286 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.317017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33025 (local address 127.1.253.66:36133)
0924 16:12:13.317151 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.317207 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 16:12:13.317755 8850 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:13.317909 8850 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:13.322444 8850 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7b93ed626234c58a2920ac0fe0dd3a7: Sending DeleteTablet for 3 replicas of tablet a968e32182ef41d9aa10f5728aadb07c
I20260924 16:12:13.323074 9422 tablet_service.cc:1514] Processing DeleteTablet for tablet a968e32182ef41d9aa10f5728aadb07c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:13 UTC) from {username='oryx', principal='oryx/127.1.253.125@KRBTEST.COM'} at 127.0.0.1:60654
I20260924 16:12:13.323184 9286 tablet_service.cc:1514] Processing DeleteTablet for tablet a968e32182ef41d9aa10f5728aadb07c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:13 UTC) from {username='oryx', principal='oryx/127.1.253.125@KRBTEST.COM'} at 127.0.0.1:54370
I20260924 16:12:13.323563 9606 tablet_replica.cc:330] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d: stopping tablet replica
I20260924 16:12:13.323685 9606 raft_consensus.cc:2229] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:13.323690 9149 tablet_service.cc:1514] Processing DeleteTablet for tablet a968e32182ef41d9aa10f5728aadb07c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:13 UTC) from {username='oryx', principal='oryx/127.1.253.125@KRBTEST.COM'} at 127.0.0.1:48842
I20260924 16:12:13.323822 9606 raft_consensus.cc:2258] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:13.324569 9606 ts_tablet_manager.cc:1905] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:13.324599 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 8996
I20260924 16:12:13.328199 9606 ts_tablet_manager.cc:1918] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:13.328290 9606 log.cc:1196] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/a968e32182ef41d9aa10f5728aadb07c
I20260924 16:12:13.328804 9606 ts_tablet_manager.cc:1939] T a968e32182ef41d9aa10f5728aadb07c P edc542371ed34c6fa9e7153ad00c242d: Deleting consensus metadata
I20260924 16:12:13.329607 9605 tablet_replica.cc:330] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1: stopping tablet replica
I20260924 16:12:13.329597 8835 catalog_manager.cc:4841] TS edc542371ed34c6fa9e7153ad00c242d (127.1.253.66:35059): tablet a968e32182ef41d9aa10f5728aadb07c (table test-table [id=23941e5d14b24776897ba03f9a2970c7]) successfully deleted
I20260924 16:12:13.329824 9605 raft_consensus.cc:2229] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:13.330137 9605 raft_consensus.cc:2258] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:13.330914 9605 ts_tablet_manager.cc:1905] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:13.334146 9605 ts_tablet_manager.cc:1918] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:13.334254 9605 log.cc:1196] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/a968e32182ef41d9aa10f5728aadb07c
I20260924 16:12:13.334575 9605 ts_tablet_manager.cc:1939] T a968e32182ef41d9aa10f5728aadb07c P dff39b9cdeb04b469a97c979003e5cf1: Deleting consensus metadata
I20260924 16:12:13.335345 8837 catalog_manager.cc:4841] TS dff39b9cdeb04b469a97c979003e5cf1 (127.1.253.67:35207): tablet a968e32182ef41d9aa10f5728aadb07c (table test-table [id=23941e5d14b24776897ba03f9a2970c7]) successfully deleted
I20260924 16:12:13.338219 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 9223
I20260924 16:12:13.338595 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.338287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40341 (local address 127.0.0.1:48858)
0924 16:12:13.338430 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:13.338497 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 16:12:13.341049 8836 catalog_manager.cc:4584] TS e2a33489ba884ef0b2eaf21e7897618b (127.1.253.65:40341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a968e32182ef41d9aa10f5728aadb07c: Network error: Client connection negotiation failed: client connection to 127.1.253.65:40341: connect: Connection refused (error 111)
I20260924 16:12:13.344848 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 9359
I20260924 16:12:13.351090 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 8715
I20260924 16:12:13.356818 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 8823
2026-09-24T16:12:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5441 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:13 dist-test-slave-6wvm krb5kdc[9610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:13 dist-test-slave-6wvm krb5kdc[9610](info): set up 2 sockets
Sep 24 16:12:13 dist-test-slave-6wvm krb5kdc[9610](info): commencing operation
krb5kdc: starting...
W20260924 16:12:14.399318 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:12:14 dist-test-slave-6wvm krb5kdc[9610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.126.keytab.
Entry for principal oryx/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.126.keytab.
I20260924 16:12:14.525173 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:33135
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.126.keytab
--principal=oryx/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:33135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:14.641340 9623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:14.641711 9623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:14.641803 9623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:14.645893 9623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:14.645990 9623 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:12:14.646019 9623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:14.646061 9623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:14.646092 9623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:14.646140 9623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:14.651566 9623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:33135
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.126.keytab
--principal=oryx/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:33135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:14.652655 9623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:14.653729 9623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:14.662721 9628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:14.662778 9623 server_base.cc:1034] running on GCE node
W20260924 16:12:14.662693 9631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:14.662688 9629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:14.663599 9623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:14.664765 9623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:14.665978 9623 hybrid_clock.cc:648] HybridClock initialized: now 1790266334665960 us; error 32 us; skew 500 ppm
Sep 24 16:12:14 dist-test-slave-6wvm krb5kdc[9610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266334, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:14.668887 9623 init.cc:377] Logged in from keytab as oryx/127.1.253.126@KRBTEST.COM (short username oryx)
I20260924 16:12:14.669871 9623 webserver.cc:460] Webserver started at http://127.1.253.126:33943/ using document root <none> and password file <none>
I20260924 16:12:14.670481 9623 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:14.670547 9623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:14.670730 9623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:14.672777 9623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "64d3b7abf07c46efab55e10d94ad40b7"
format_stamp: "Formatted at 2026-09-24 16:12:14 on dist-test-slave-6wvm"
server_key: "ea4237be12c0d6bc25c575e79fbd4e2f"
server_key_iv: "add3ecb4f43a4cf73fe56fd409eac4b1"
server_key_version: "encryptionkey@0"
I20260924 16:12:14.673316 9623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "64d3b7abf07c46efab55e10d94ad40b7"
format_stamp: "Formatted at 2026-09-24 16:12:14 on dist-test-slave-6wvm"
server_key: "ea4237be12c0d6bc25c575e79fbd4e2f"
server_key_iv: "add3ecb4f43a4cf73fe56fd409eac4b1"
server_key_version: "encryptionkey@0"
I20260924 16:12:14.677206 9623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:12:14.679803 9638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:14.681133 9623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:12:14.681252 9623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "64d3b7abf07c46efab55e10d94ad40b7"
format_stamp: "Formatted at 2026-09-24 16:12:14 on dist-test-slave-6wvm"
server_key: "ea4237be12c0d6bc25c575e79fbd4e2f"
server_key_iv: "add3ecb4f43a4cf73fe56fd409eac4b1"
server_key_version: "encryptionkey@0"
I20260924 16:12:14.681347 9623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:14.710446 9623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:14.713999 9623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:14.714161 9623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:14.722755 9623 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:33135
I20260924 16:12:14.722785 9700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:33135 every 8 connection(s)
I20260924 16:12:14.723814 9623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:14.726790 9701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:14.732535 9701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7: Bootstrap starting.
I20260924 16:12:14.732928 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 9623
I20260924 16:12:14.733023 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:14.733353 2037 external_mini_cluster.cc:1428] Setting key c0681d9438eafc960fef5fcdb5976405
I20260924 16:12:14.735213 9701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:14.735960 9701 log.cc:824] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:14.737862 9701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7: No bootstrap required, opened a new log
I20260924 16:12:14.741505 9701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d3b7abf07c46efab55e10d94ad40b7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33135 } }
I20260924 16:12:14.741791 9701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:14.741881 9701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d3b7abf07c46efab55e10d94ad40b7, State: Initialized, Role: FOLLOWER
Sep 24 16:12:14 dist-test-slave-6wvm krb5kdc[9610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
I20260924 16:12:14.742395 9701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d3b7abf07c46efab55e10d94ad40b7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33135 } }
I20260924 16:12:14.742545 9701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:14.742631 9701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:14.742735 9701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:14.743906 9701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d3b7abf07c46efab55e10d94ad40b7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33135 } }
I20260924 16:12:14.744290 9701 leader_election.cc:302] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64d3b7abf07c46efab55e10d94ad40b7; no voters:
I20260924 16:12:14.744625 9701 leader_election.cc:288] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:14.744740 9706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:14.745008 9706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [term 1 LEADER]: Becoming Leader. State: Replica: 64d3b7abf07c46efab55e10d94ad40b7, State: Running, Role: LEADER
I20260924 16:12:14.745386 9706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d3b7abf07c46efab55e10d94ad40b7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33135 } }
I20260924 16:12:14.745962 9701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:14.748713 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:14.734713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.1.253.126:33135)
0924 16:12:14.735137 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:14.735154 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:14.735190 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:12:14.735850 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:14.735868 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:14.736263 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:14.736605 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:14.737459 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.738559 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.739410 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.739882 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.740053 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:14.742760 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:14.742766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:14.742783 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:14.742819 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:14.744828 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:14.745348 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:14.745354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:14.745360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:14.745497 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:14.745790 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:14.745793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:14.745795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:14.746320 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:14.746532 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:14.747326 (+ 794us) server_negotiation.cc:299] Negotiation successful
0924 16:12:14.747655 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":130,"threads_started":1}
I20260924 16:12:14.748814 9707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64d3b7abf07c46efab55e10d94ad40b7. Latest consensus state: current_term: 1 leader_uuid: "64d3b7abf07c46efab55e10d94ad40b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d3b7abf07c46efab55e10d94ad40b7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33135 } } }
I20260924 16:12:14.748978 9707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:14.748991 9708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64d3b7abf07c46efab55e10d94ad40b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d3b7abf07c46efab55e10d94ad40b7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 33135 } } }
I20260924 16:12:14.749094 9708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7 [sys.catalog]: This master's current role is: LEADER
W20260924 16:12:14.752063 9721 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:12:14.752141 9721 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:12:14.752378 9722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:14.753293 9722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:14.759693 9722 catalog_manager.cc:1347] Generated new cluster ID: 65a0f62d8d4f4000bb8beaa465e24440
I20260924 16:12:14.759774 9722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:14.768499 9722 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:14.769381 9722 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:14.778741 9722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7: Generated new TSK 0
I20260924 16:12:14.779443 9722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab.
Entry for principal oryx/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab.
I20260924 16:12:14.809911 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:33135
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab
--principal=oryx/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:14.925247 9727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:14.925499 9727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:14.925541 9727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:14.929448 9727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:14.929497 9727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:14.929535 9727 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:12:14.929567 9727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:12:14.934600 9727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab
--principal=oryx/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.253.126:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:14.935528 9727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:14.936517 9727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:14.945817 9732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:14.945817 9735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:14.945919 9733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:14.946017 9727 server_base.cc:1034] running on GCE node
I20260924 16:12:14.946565 9727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:14.947134 9727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:14.948315 9727 hybrid_clock.cc:648] HybridClock initialized: now 1790266334948283 us; error 51 us; skew 500 ppm
Sep 24 16:12:14 dist-test-slave-6wvm krb5kdc[9610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266334, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:14.951159 9727 init.cc:377] Logged in from keytab as oryx/127.1.253.65@KRBTEST.COM (short username oryx)
I20260924 16:12:14.951999 9727 webserver.cc:460] Webserver started at http://127.1.253.65:33321/ using document root <none> and password file <none>
I20260924 16:12:14.952575 9727 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:14.952633 9727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:14.952862 9727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:14.954867 9727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "061877b5d65e403d898b7d580cf6f9d1"
format_stamp: "Formatted at 2026-09-24 16:12:14 on dist-test-slave-6wvm"
server_key: "b82fa3f0d71a784fc9dad21e5fe2753a"
server_key_iv: "b7f91f7dfa84188774778a1f17a82632"
server_key_version: "encryptionkey@0"
I20260924 16:12:14.955403 9727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "061877b5d65e403d898b7d580cf6f9d1"
format_stamp: "Formatted at 2026-09-24 16:12:14 on dist-test-slave-6wvm"
server_key: "b82fa3f0d71a784fc9dad21e5fe2753a"
server_key_iv: "b7f91f7dfa84188774778a1f17a82632"
server_key_version: "encryptionkey@0"
I20260924 16:12:14.959044 9727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:14.961364 9742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:14.962589 9727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:14.962714 9727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "061877b5d65e403d898b7d580cf6f9d1"
format_stamp: "Formatted at 2026-09-24 16:12:14 on dist-test-slave-6wvm"
server_key: "b82fa3f0d71a784fc9dad21e5fe2753a"
server_key_iv: "b7f91f7dfa84188774778a1f17a82632"
server_key_version: "encryptionkey@0"
I20260924 16:12:14.962849 9727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:14.986325 9727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:14.989194 9727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:14.989396 9727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:14.990875 9727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:14.990950 9727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:14.991022 9727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:14.991073 9727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:12:14 dist-test-slave-6wvm krb5kdc[9610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266334, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
I20260924 16:12:15.002341 9727 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:37359
I20260924 16:12:15.002425 9857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:37359 every 8 connection(s)
I20260924 16:12:15.003546 9727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:15.005228 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:14.992017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33135 (local address 127.1.253.65:55793)
0924 16:12:14.992493 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:14.992542 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:14.993458 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:14.994363 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:14.994370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:14.994705 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:14.995306 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:14.995323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.996305 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.996309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:14.997475 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:14.997485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.998053 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.998057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:14.998299 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:14.998936 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:14.998951 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:15.000552 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:15.002446 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.002452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.002467 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.002804 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.003083 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.003086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.003088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.003258 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.003637 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:15.003642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:15.003963 (+ 321us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.004181 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.004490 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":147,"threads_started":1}
I20260924 16:12:15.005759 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:14.991650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:55793 (local address 127.1.253.126:33135)
0924 16:12:14.991820 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:14.991827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:14.992651 (+ 824us) server_negotiation.cc:407] Connection header received
0924 16:12:14.993747 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:14.993750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:14.993831 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:14.993895 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:14.995467 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.996176 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.997603 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.997887 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.997996 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.000737 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.000740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.000743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.000772 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.002320 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.002903 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.002906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.002908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.002988 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.003363 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.003366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.003367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.003533 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.003633 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.005519 (+ 1886us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.005641 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:12:15.006665 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 9727
I20260924 16:12:15.006804 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:15.007000 9858 heartbeater.cc:347] Connected to a master server at 127.1.253.126:33135
I20260924 16:12:15.007158 2037 external_mini_cluster.cc:1428] Setting key 920589dafd305265e3f0f83475c85f10
I20260924 16:12:15.007205 9858 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:15.007802 9858 heartbeater.cc:511] Master 127.1.253.126:33135 requested a full tablet report, sending...
I20260924 16:12:15.009529 9654 ts_manager.cc:194] Registered new tserver with Master: 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359)
I20260924 16:12:15.011153 9654 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:55793
I20260924 16:12:15.017293 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.009435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:56885 (local address 127.1.253.126:33135)
0924 16:12:15.009562 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.009568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.009744 (+ 176us) server_negotiation.cc:407] Connection header received
0924 16:12:15.009862 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.009868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.009924 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.010018 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.010548 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.011284 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.012427 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.012713 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.012834 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.014052 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.014056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.014059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.014083 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.015677 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.016116 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.016120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.016122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.016203 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.016480 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.016485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.016489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.016661 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.016767 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.017029 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.017132 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 16:12:15.017302 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.009336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33135 (local address 127.1.253.65:56885)
0924 16:12:15.009659 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.009686 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.009783 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.010056 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.010059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.010286 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:15.010412 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.010420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.011443 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.011446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.012297 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.012312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.012841 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.012844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.012938 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.013496 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:15.013517 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:15.013908 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:15.015808 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.015811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.015813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.016015 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.016296 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.016299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.016301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.016390 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.016758 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:15.016761 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:15.016860 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.016967 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.017112 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.1.253.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab.
Entry for principal oryx/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab.
I20260924 16:12:15.036805 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:33135
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab
--principal=oryx/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:15.149256 9867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:15.149533 9867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:15.149614 9867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:15.153546 9867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:15.153625 9867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:15.153693 9867 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:12:15.153774 9867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:12:15.158910 9867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab
--principal=oryx/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.253.126:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:15.159946 9867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:15.161015 9867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:15.170704 9872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:15.170679 9867 server_base.cc:1034] running on GCE node
W20260924 16:12:15.170668 9873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:15.170687 9875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:15.171334 9867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:15.171954 9867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:15.173139 9867 hybrid_clock.cc:648] HybridClock initialized: now 1790266335173119 us; error 30 us; skew 500 ppm
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:15.175850 9867 init.cc:377] Logged in from keytab as oryx/127.1.253.66@KRBTEST.COM (short username oryx)
I20260924 16:12:15.176705 9867 webserver.cc:460] Webserver started at http://127.1.253.66:34731/ using document root <none> and password file <none>
I20260924 16:12:15.177235 9867 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:15.177294 9867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:15.177460 9867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:15.179518 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "74c74b33aa9840b3b6df3c4381f6b454"
format_stamp: "Formatted at 2026-09-24 16:12:15 on dist-test-slave-6wvm"
server_key: "e4c48406e8b9cbd0fc66bcec1135de1a"
server_key_iv: "3fc3ba5424f101eff31accebbcd6e589"
server_key_version: "encryptionkey@0"
I20260924 16:12:15.180240 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "74c74b33aa9840b3b6df3c4381f6b454"
format_stamp: "Formatted at 2026-09-24 16:12:15 on dist-test-slave-6wvm"
server_key: "e4c48406e8b9cbd0fc66bcec1135de1a"
server_key_iv: "3fc3ba5424f101eff31accebbcd6e589"
server_key_version: "encryptionkey@0"
I20260924 16:12:15.184134 9867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:12:15.186817 9882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:15.188028 9867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:15.188124 9867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "74c74b33aa9840b3b6df3c4381f6b454"
format_stamp: "Formatted at 2026-09-24 16:12:15 on dist-test-slave-6wvm"
server_key: "e4c48406e8b9cbd0fc66bcec1135de1a"
server_key_iv: "3fc3ba5424f101eff31accebbcd6e589"
server_key_version: "encryptionkey@0"
I20260924 16:12:15.188203 9867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:15.214123 9867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:15.217445 9867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:15.217603 9867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:15.219146 9867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:15.219199 9867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:15.219241 9867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:15.219291 9867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
I20260924 16:12:15.230551 9867 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:45527
I20260924 16:12:15.230623 9997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:45527 every 8 connection(s)
I20260924 16:12:15.231670 9867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:15.233414 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 9867
I20260924 16:12:15.233561 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:15.233826 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.219937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:50569 (local address 127.1.253.126:33135)
0924 16:12:15.220124 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.220132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.220975 (+ 843us) server_negotiation.cc:407] Connection header received
0924 16:12:15.222005 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.222009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.222064 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.222125 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.223920 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.224698 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.226001 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.226326 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.226428 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.229517 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.229521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.229524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.229552 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.231326 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.232048 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.232052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.232054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.232192 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.232664 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.232667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.232669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.232899 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.233001 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.233493 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.233641 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:12:15.233907 2037 external_mini_cluster.cc:1428] Setting key ceeeae2cc293e1fad64c96c63b1ff430
I20260924 16:12:15.234670 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.220352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33135 (local address 127.1.253.66:50569)
0924 16:12:15.220808 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.220858 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.221742 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.222645 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.222654 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.223145 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:15.223717 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.223742 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.224811 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.224815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.225872 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.225882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.226485 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.226489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.226657 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.227355 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:15.227398 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:15.229319 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:15.231496 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.231501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.231513 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.231933 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.232323 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.232327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.232331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.232544 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.232995 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:15.233000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:15.233221 (+ 221us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.233386 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.233708 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":125,"threads_started":1}
I20260924 16:12:15.236088 9998 heartbeater.cc:347] Connected to a master server at 127.1.253.126:33135
I20260924 16:12:15.236222 9998 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:15.236785 9998 heartbeater.cc:511] Master 127.1.253.126:33135 requested a full tablet report, sending...
I20260924 16:12:15.238020 9655 ts_manager.cc:194] Registered new tserver with Master: 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66:45527)
I20260924 16:12:15.238662 9655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:50569
WARNING: no policy specified for oryx/127.1.253.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.67@KRBTEST.COM" created.
I20260924 16:12:15.250540 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.242579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33135 (local address 127.1.253.66:35461)
0924 16:12:15.242831 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.242852 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.242933 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.243197 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.243200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.243360 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:15.243450 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.243455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.244555 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.244557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.245100 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.245107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.245753 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.245756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.245830 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.246433 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:15.246445 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:15.246821 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:15.248899 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.248904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.248907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.249120 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.249427 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.249431 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.249433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.249530 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.249939 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:15.249942 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:15.250057 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.250173 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.250337 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":97,"threads_started":1}
I20260924 16:12:15.250532 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.242656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:35461 (local address 127.1.253.126:33135)
0924 16:12:15.242788 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.242796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.242926 (+ 130us) server_negotiation.cc:407] Connection header received
0924 16:12:15.243006 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.243011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.243073 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.243175 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.243574 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.244417 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.245223 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.245626 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.245739 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.246955 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.246958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.246961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.247004 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.248764 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.249234 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.249240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.249243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.249339 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.249626 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.249631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.249632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.249836 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.249957 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.250228 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.250370 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab.
Entry for principal oryx/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab.
I20260924 16:12:15.265318 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:33135
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab
--principal=oryx/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:15.377669 10007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:15.377934 10007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:15.378014 10007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:15.382076 10007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:15.382145 10007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:15.382207 10007 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:12:15.382308 10007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:12:15.387741 10007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab
--principal=oryx/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.253.126:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:15.388758 10007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:15.390060 10007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:15.400367 10012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:15.400362 10013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:15.400520 10007 server_base.cc:1034] running on GCE node
W20260924 16:12:15.400521 10015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:15.401086 10007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:15.401813 10007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:15.402977 10007 hybrid_clock.cc:648] HybridClock initialized: now 1790266335402969 us; error 30 us; skew 500 ppm
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:15.405637 10007 init.cc:377] Logged in from keytab as oryx/127.1.253.67@KRBTEST.COM (short username oryx)
I20260924 16:12:15.406533 10007 webserver.cc:460] Webserver started at http://127.1.253.67:45903/ using document root <none> and password file <none>
I20260924 16:12:15.407125 10007 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:15.407189 10007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:15.407424 10007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:15.409510 10007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "254a0a166d2f4261b2fc92503c87e432"
format_stamp: "Formatted at 2026-09-24 16:12:15 on dist-test-slave-6wvm"
server_key: "45a50b4e5a0d49be0c1a6a081a716203"
server_key_iv: "1ec576da7eb36e26ee0620273f9f5af5"
server_key_version: "encryptionkey@0"
I20260924 16:12:15.410056 10007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "254a0a166d2f4261b2fc92503c87e432"
format_stamp: "Formatted at 2026-09-24 16:12:15 on dist-test-slave-6wvm"
server_key: "45a50b4e5a0d49be0c1a6a081a716203"
server_key_iv: "1ec576da7eb36e26ee0620273f9f5af5"
server_key_version: "encryptionkey@0"
I20260924 16:12:15.413866 10007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:15.416330 10022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:15.417536 10007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:15.417661 10007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "254a0a166d2f4261b2fc92503c87e432"
format_stamp: "Formatted at 2026-09-24 16:12:15 on dist-test-slave-6wvm"
server_key: "45a50b4e5a0d49be0c1a6a081a716203"
server_key_iv: "1ec576da7eb36e26ee0620273f9f5af5"
server_key_version: "encryptionkey@0"
I20260924 16:12:15.417773 10007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:15.441962 10007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:15.445636 10007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:15.445852 10007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:15.447667 10007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:15.447746 10007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:15.447820 10007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:15.447876 10007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
I20260924 16:12:15.459200 10007 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:43113
I20260924 16:12:15.459246 10137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:43113 every 8 connection(s)
I20260924 16:12:15.460261 10007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:15.462195 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 10007
I20260924 16:12:15.462380 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:15.462672 2037 external_mini_cluster.cc:1428] Setting key 6f8f21647027639426304022305b4829
I20260924 16:12:15.462669 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.448568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:52161 (local address 127.1.253.126:33135)
0924 16:12:15.448717 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.448721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.449890 (+ 1169us) server_negotiation.cc:407] Connection header received
0924 16:12:15.451098 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.451102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.451171 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.451293 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.452907 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.453800 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.455284 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.455534 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.455661 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.458636 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.458640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.458643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.458681 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.460417 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.461028 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.461032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.461034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.461132 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.461521 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.461525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.461526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.461729 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.461815 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.462347 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.462493 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:12:15.463519 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.449144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33135 (local address 127.1.253.67:52161)
0924 16:12:15.449717 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.449760 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.450797 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.451486 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.451498 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.452021 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:15.452737 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.452761 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.453923 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.453928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.455145 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.455155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.455716 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.455720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.455891 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.456554 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:15.456571 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:15.458449 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:15.460550 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.460563 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.460575 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.460892 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.461243 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.461246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.461248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.461409 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.461829 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:15.461834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:15.462070 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.462273 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.462599 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":134,"threads_started":1}
I20260924 16:12:15.464980 10138 heartbeater.cc:347] Connected to a master server at 127.1.253.126:33135
I20260924 16:12:15.465129 10138 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:15.465695 10138 heartbeater.cc:511] Master 127.1.253.126:33135 requested a full tablet report, sending...
I20260924 16:12:15.466778 9655 ts_manager.cc:194] Registered new tserver with Master: 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67:43113)
I20260924 16:12:15.467594 9655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:52161
I20260924 16:12:15.473815 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.467263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:42067 (local address 127.1.253.126:33135)
0924 16:12:15.467403 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.467408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.467535 (+ 127us) server_negotiation.cc:407] Connection header received
0924 16:12:15.467653 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.467655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.467735 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.467810 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.468289 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.468785 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.469595 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.469820 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.469925 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.471022 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.471025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.471028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.471061 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.472249 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.472667 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.472671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.472672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.472756 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.473031 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.473035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.473037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.473204 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.473312 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.473574 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.473679 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:12:15.473827 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.467141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33135 (local address 127.1.253.67:42067)
0924 16:12:15.467439 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.467453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.467561 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.467830 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.467833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.468066 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:15.468172 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.468178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.468908 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.468911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.469478 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.469486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.469930 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.469933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.470031 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.470547 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:15.470558 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:15.470844 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:15.472360 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.472365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.472367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.472567 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.472855 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.472858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.472860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.472941 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.473303 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:15.473306 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:15.473404 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.473511 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.473648 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":100,"threads_started":1}
I20260924 16:12:15.476851 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:12:15.500941 9704 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:15.494986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.1.253.126:33135)
0924 16:12:15.495130 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.495134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.495148 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:12:15.495190 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.495192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.495234 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.495307 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.495992 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.496515 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.497324 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.497526 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.497614 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.500773 (+ 3159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60148: BlockingRecv error: recv got EOF from 127.0.0.1:60148 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
I20260924 16:12:15.510003 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.502559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.1.253.126:33135)
0924 16:12:15.502681 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.502684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.502844 (+ 160us) server_negotiation.cc:407] Connection header received
0924 16:12:15.502949 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.502951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.502988 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.503054 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.503594 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.504074 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.504761 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.504951 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.505025 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.507287 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.507291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.507293 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.507315 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.508736 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.509085 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.509088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.509089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.509158 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.509386 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.509388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.509389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.509547 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.509646 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.509806 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.509889 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:12:15.513100 9655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:15.516664 9655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:15.531425 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.526190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:45527 (local address 127.0.0.1:44526)
0924 16:12:15.526844 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.526857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.526958 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.527539 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.527542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.527559 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.527751 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.527756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.528749 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.528754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.529978 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.529987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.530881 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.530885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.530998 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.531037 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.531157 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.531282 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":22,"spinlock_wait_cycles":5376,"thread_start_us":125,"threads_started":1}
I20260924 16:12:15.531435 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.526450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.1.253.66:45527)
0924 16:12:15.526787 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.526801 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.527031 (+ 230us) server_negotiation.cc:407] Connection header received
0924 16:12:15.527105 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.527109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.527259 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.527385 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.527886 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.528625 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.530093 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.530754 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.530882 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.531052 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.531196 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.531284 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":120,"threads_started":1}
I20260924 16:12:15.531829 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.525714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:43113 (local address 127.0.0.1:59540)
0924 16:12:15.526527 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.526541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.526636 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.527522 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.527524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.527538 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.527748 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.527756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.529159 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.529163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.530146 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.530155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.531443 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.531448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.531542 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.531564 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.531633 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.531697 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":71,"thread_start_us":118,"threads_started":1}
I20260924 16:12:15.532243 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.526286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59540 (local address 127.1.253.67:43113)
0924 16:12:15.527101 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.527110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.527129 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:12:15.527188 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.527191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.527375 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.527631 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.527871 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.529015 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.530358 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.531138 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.531237 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.531383 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.532002 (+ 619us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.532107 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":112,"threads_started":1}
I20260924 16:12:15.532758 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.525478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:37359 (local address 127.0.0.1:33728)
0924 16:12:15.525842 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.525895 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.526011 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.527389 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.527395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.527428 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.527662 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.527671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.529159 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.529163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.531352 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.531364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.532412 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.532419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.532508 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.532532 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.532605 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.532649 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1098,"thread_start_us":62,"threads_started":1}
I20260924 16:12:15.533335 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.525628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.1.253.65:37359)
0924 16:12:15.526861 (+ 1233us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.526867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.526895 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:12:15.527001 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.527010 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.527228 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.527386 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.527828 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.529015 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.531488 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.532272 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.532953 (+ 681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.533085 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.533135 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.533217 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1142,"thread_start_us":85,"threads_started":1}
I20260924 16:12:15.535120 9932 tablet_service.cc:1467] Processing CreateTablet for tablet a6749bdeceb249869e70d095bd1824d3 (DEFAULT_TABLE table=test-table [id=72976c9c34bd42f5822013c30230449d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.535310 10072 tablet_service.cc:1467] Processing CreateTablet for tablet a6749bdeceb249869e70d095bd1824d3 (DEFAULT_TABLE table=test-table [id=72976c9c34bd42f5822013c30230449d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.536288 9932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6749bdeceb249869e70d095bd1824d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.536288 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6749bdeceb249869e70d095bd1824d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.543649 9792 tablet_service.cc:1467] Processing CreateTablet for tablet a6749bdeceb249869e70d095bd1824d3 (DEFAULT_TABLE table=test-table [id=72976c9c34bd42f5822013c30230449d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.544265 10166 tablet_bootstrap.cc:492] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454: Bootstrap starting.
I20260924 16:12:15.544593 9792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6749bdeceb249869e70d095bd1824d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.545415 10167 tablet_bootstrap.cc:492] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432: Bootstrap starting.
I20260924 16:12:15.546662 10166 tablet_bootstrap.cc:654] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.547304 10167 tablet_bootstrap.cc:654] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.547660 10166 log.cc:824] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:15.548167 10167 log.cc:824] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:15.549660 10166 tablet_bootstrap.cc:492] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454: No bootstrap required, opened a new log
I20260924 16:12:15.549846 10166 ts_tablet_manager.cc:1397] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:12:15.550146 10167 tablet_bootstrap.cc:492] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432: No bootstrap required, opened a new log
I20260924 16:12:15.550336 10167 ts_tablet_manager.cc:1397] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 16:12:15.551159 10170 tablet_bootstrap.cc:492] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1: Bootstrap starting.
I20260924 16:12:15.553040 10170 tablet_bootstrap.cc:654] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.553759 10170 log.cc:824] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:15.553838 10166 raft_consensus.cc:348] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.554100 10167 raft_consensus.cc:348] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.554363 10167 raft_consensus.cc:374] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.554459 10167 raft_consensus.cc:729] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254a0a166d2f4261b2fc92503c87e432, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.554637 10166 raft_consensus.cc:374] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.554767 10166 raft_consensus.cc:729] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c74b33aa9840b3b6df3c4381f6b454, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.554981 10167 consensus_queue.cc:260] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.555254 10166 consensus_queue.cc:260] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.555652 10138 heartbeater.cc:503] Master 127.1.253.126:33135 was elected leader, sending a full tablet report...
I20260924 16:12:15.555776 10170 tablet_bootstrap.cc:492] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1: No bootstrap required, opened a new log
I20260924 16:12:15.555943 10170 ts_tablet_manager.cc:1397] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:12:15.556017 10167 ts_tablet_manager.cc:1428] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:12:15.556318 10166 ts_tablet_manager.cc:1428] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:12:15.556617 9998 heartbeater.cc:503] Master 127.1.253.126:33135 was elected leader, sending a full tablet report...
I20260924 16:12:15.561322 10170 raft_consensus.cc:348] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.561568 10170 raft_consensus.cc:374] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.561628 10170 raft_consensus.cc:729] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061877b5d65e403d898b7d580cf6f9d1, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.562013 10170 consensus_queue.cc:260] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.562712 9858 heartbeater.cc:503] Master 127.1.253.126:33135 was elected leader, sending a full tablet report...
I20260924 16:12:15.563056 10170 ts_tablet_manager.cc:1428] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 16:12:15.615726 10173 raft_consensus.cc:482] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:15.615947 10173 raft_consensus.cc:504] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.617129 10173 leader_election.cc:288] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359), 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67:43113)
I20260924 16:12:15.620887 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.617370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:37359 (local address 127.1.253.66:59525)
0924 16:12:15.617593 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.617610 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.617710 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.617991 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.617994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.618011 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.618130 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.618136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.618939 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.618945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.619721 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.619730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.620419 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.620423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.620547 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.620576 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.620669 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.620754 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260924 16:12:15.620918 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.617500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:59525 (local address 127.1.253.65:37359)
0924 16:12:15.617691 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.617696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.617712 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:12:15.617839 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.617844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.617892 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.617978 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.618272 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.618823 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.619860 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.620289 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.620427 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.620510 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.620743 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.620814 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 16:12:15.621574 9812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6749bdeceb249869e70d095bd1824d3" candidate_uuid: "74c74b33aa9840b3b6df3c4381f6b454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "061877b5d65e403d898b7d580cf6f9d1" is_pre_election: true
I20260924 16:12:15.621833 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.617390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:43113 (local address 127.1.253.66:42801)
0924 16:12:15.617808 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.617820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.617943 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.618216 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.618219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.618261 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.618385 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.618390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.619148 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.619153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.620397 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.620407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.621400 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.621405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.621509 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.621540 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.621636 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.621701 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":134,"threads_started":1}
I20260924 16:12:15.621860 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.617522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:42801 (local address 127.1.253.67:43113)
0924 16:12:15.617670 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.617677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.617905 (+ 228us) server_negotiation.cc:407] Connection header received
0924 16:12:15.618075 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.618079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.618119 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.618204 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.618498 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.619031 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.620582 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.621147 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.621259 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.621319 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.621654 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.621725 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:12:15.621954 9812 raft_consensus.cc:2454] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c74b33aa9840b3b6df3c4381f6b454 in term 0.
I20260924 16:12:15.622517 9885 leader_election.cc:302] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 061877b5d65e403d898b7d580cf6f9d1, 74c74b33aa9840b3b6df3c4381f6b454; no voters:
I20260924 16:12:15.622519 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6749bdeceb249869e70d095bd1824d3" candidate_uuid: "74c74b33aa9840b3b6df3c4381f6b454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254a0a166d2f4261b2fc92503c87e432" is_pre_election: true
I20260924 16:12:15.622797 10092 raft_consensus.cc:2454] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c74b33aa9840b3b6df3c4381f6b454 in term 0.
I20260924 16:12:15.622809 10173 raft_consensus.cc:2781] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:15.622916 10173 raft_consensus.cc:482] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:15.622994 10173 raft_consensus.cc:3037] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.624074 10173 raft_consensus.cc:504] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.624472 10173 leader_election.cc:288] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [CANDIDATE]: Term 1 election: Requested vote from peers 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359), 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67:43113)
I20260924 16:12:15.624838 9812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6749bdeceb249869e70d095bd1824d3" candidate_uuid: "74c74b33aa9840b3b6df3c4381f6b454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "061877b5d65e403d898b7d580cf6f9d1"
I20260924 16:12:15.624853 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6749bdeceb249869e70d095bd1824d3" candidate_uuid: "74c74b33aa9840b3b6df3c4381f6b454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254a0a166d2f4261b2fc92503c87e432"
I20260924 16:12:15.624941 9812 raft_consensus.cc:3037] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.624994 10092 raft_consensus.cc:3037] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.625989 9812 raft_consensus.cc:2454] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c74b33aa9840b3b6df3c4381f6b454 in term 1.
I20260924 16:12:15.626328 10092 raft_consensus.cc:2454] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c74b33aa9840b3b6df3c4381f6b454 in term 1.
I20260924 16:12:15.626343 9885 leader_election.cc:302] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 061877b5d65e403d898b7d580cf6f9d1, 74c74b33aa9840b3b6df3c4381f6b454; no voters:
I20260924 16:12:15.626560 10173 raft_consensus.cc:2781] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:15.626806 10173 raft_consensus.cc:686] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 LEADER]: Becoming Leader. State: Replica: 74c74b33aa9840b3b6df3c4381f6b454, State: Running, Role: LEADER
I20260924 16:12:15.627162 10173 consensus_queue.cc:237] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.630450 9653 catalog_manager.cc:5495] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66). New cstate: current_term: 1 leader_uuid: "74c74b33aa9840b3b6df3c4381f6b454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } health_report { overall_health: UNKNOWN } } }
I20260924 16:12:15.653916 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.650434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.1.253.66:45527)
0924 16:12:15.650610 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.650614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.650629 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:12:15.650705 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.650707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.650763 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.650893 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:15.651146 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.651691 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.652512 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.652729 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.652805 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.653035 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:15.653509 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:15.653599 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.653659 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.653710 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:12:15.660001 10092 raft_consensus.cc:1261] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Refusing update from remote peer 74c74b33aa9840b3b6df3c4381f6b454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:15.660002 9812 raft_consensus.cc:1261] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Refusing update from remote peer 74c74b33aa9840b3b6df3c4381f6b454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:15.660746 10173 consensus_queue.cc:1035] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:15.660967 10176 consensus_queue.cc:1035] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:15.670701 10181 mvcc.cc:205] Tried to move back new op lower bound from 7332930910855319552 to 7332930910731608064. Current Snapshot: MvccSnapshot[applied={T|T < 7332930910855319552}]
I20260924 16:12:15.670885 10180 mvcc.cc:205] Tried to move back new op lower bound from 7332930910855319552 to 7332930910731608064. Current Snapshot: MvccSnapshot[applied={T|T < 7332930910855319552}]
I20260924 16:12:15.676302 9653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:15.676445 9653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:15.678841 9653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7: Sending DeleteTablet for 3 replicas of tablet a6749bdeceb249869e70d095bd1824d3
I20260924 16:12:15.679677 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet a6749bdeceb249869e70d095bd1824d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:15 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:59540
I20260924 16:12:15.680047 9792 tablet_service.cc:1514] Processing DeleteTablet for tablet a6749bdeceb249869e70d095bd1824d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:15 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:33728
I20260924 16:12:15.680285 10188 tablet_replica.cc:330] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432: stopping tablet replica
I20260924 16:12:15.680368 9932 tablet_service.cc:1514] Processing DeleteTablet for tablet a6749bdeceb249869e70d095bd1824d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:15 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:44526
I20260924 16:12:15.680734 10188 raft_consensus.cc:2229] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:15.680843 10189 tablet_replica.cc:330] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1: stopping tablet replica
I20260924 16:12:15.680804 9653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:12:15.681164 10189 raft_consensus.cc:2229] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 16:12:15.681226 9653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:15.681424 10189 raft_consensus.cc:2258] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:15.681485 10188 raft_consensus.cc:2258] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:15.681798 10190 tablet_replica.cc:330] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454: stopping tablet replica
I20260924 16:12:15.682061 10190 raft_consensus.cc:2229] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:15.682477 10190 raft_consensus.cc:2258] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:15.683322 10188 ts_tablet_manager.cc:1905] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:15.683751 10190 ts_tablet_manager.cc:1905] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:15.686447 10188 ts_tablet_manager.cc:1918] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:15.686627 10188 log.cc:1196] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/a6749bdeceb249869e70d095bd1824d3
I20260924 16:12:15.686880 10072 tablet_service.cc:1467] Processing CreateTablet for tablet b555ec56dd274f4895203a77d7b0bef2 (DEFAULT_TABLE table=test-table [id=4b0f972ecfe34bd78f94ca2cc4de4d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.686944 10190 ts_tablet_manager.cc:1918] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:15.687052 10190 log.cc:1196] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/a6749bdeceb249869e70d095bd1824d3
I20260924 16:12:15.687150 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b555ec56dd274f4895203a77d7b0bef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.687229 10188 ts_tablet_manager.cc:1939] T a6749bdeceb249869e70d095bd1824d3 P 254a0a166d2f4261b2fc92503c87e432: Deleting consensus metadata
I20260924 16:12:15.687362 10190 ts_tablet_manager.cc:1939] T a6749bdeceb249869e70d095bd1824d3 P 74c74b33aa9840b3b6df3c4381f6b454: Deleting consensus metadata
I20260924 16:12:15.688591 9640 catalog_manager.cc:4841] TS 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67:43113): tablet a6749bdeceb249869e70d095bd1824d3 (table test-table [id=72976c9c34bd42f5822013c30230449d]) successfully deleted
I20260924 16:12:15.689342 10167 tablet_bootstrap.cc:492] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432: Bootstrap starting.
I20260924 16:12:15.689545 9641 catalog_manager.cc:4841] TS 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66:45527): tablet a6749bdeceb249869e70d095bd1824d3 (table test-table [id=72976c9c34bd42f5822013c30230449d]) successfully deleted
I20260924 16:12:15.690151 9792 tablet_service.cc:1467] Processing CreateTablet for tablet b555ec56dd274f4895203a77d7b0bef2 (DEFAULT_TABLE table=test-table [id=4b0f972ecfe34bd78f94ca2cc4de4d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.690231 9932 tablet_service.cc:1467] Processing CreateTablet for tablet b555ec56dd274f4895203a77d7b0bef2 (DEFAULT_TABLE table=test-table [id=4b0f972ecfe34bd78f94ca2cc4de4d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.690395 10167 tablet_bootstrap.cc:654] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.690492 9932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b555ec56dd274f4895203a77d7b0bef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.690498 9792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b555ec56dd274f4895203a77d7b0bef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.692780 10170 tablet_bootstrap.cc:492] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1: Bootstrap starting.
I20260924 16:12:15.692852 10166 tablet_bootstrap.cc:492] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454: Bootstrap starting.
I20260924 16:12:15.693653 10170 tablet_bootstrap.cc:654] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.693822 10166 tablet_bootstrap.cc:654] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.694996 10170 tablet_bootstrap.cc:492] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1: No bootstrap required, opened a new log
I20260924 16:12:15.695111 10170 ts_tablet_manager.cc:1397] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:15.695381 10166 tablet_bootstrap.cc:492] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454: No bootstrap required, opened a new log
I20260924 16:12:15.695468 10166 ts_tablet_manager.cc:1397] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:15.695616 10170 raft_consensus.cc:348] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.695737 10170 raft_consensus.cc:374] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.695778 10170 raft_consensus.cc:729] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061877b5d65e403d898b7d580cf6f9d1, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.695927 10170 consensus_queue.cc:260] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.695916 10166 raft_consensus.cc:348] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.696017 10166 raft_consensus.cc:374] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.696046 10166 raft_consensus.cc:729] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c74b33aa9840b3b6df3c4381f6b454, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.696178 10166 consensus_queue.cc:260] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.696233 10170 ts_tablet_manager.cc:1428] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 16:12:15.696449 10166 ts_tablet_manager.cc:1428] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:15.699440 10189 ts_tablet_manager.cc:1905] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:15.702836 10189 ts_tablet_manager.cc:1918] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:15.702944 10189 log.cc:1196] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/wal/wals/a6749bdeceb249869e70d095bd1824d3
I20260924 16:12:15.703253 10189 ts_tablet_manager.cc:1939] T a6749bdeceb249869e70d095bd1824d3 P 061877b5d65e403d898b7d580cf6f9d1: Deleting consensus metadata
I20260924 16:12:15.704577 9641 catalog_manager.cc:4841] TS 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359): tablet a6749bdeceb249869e70d095bd1824d3 (table test-table [id=72976c9c34bd42f5822013c30230449d]) successfully deleted
I20260924 16:12:15.705286 10167 tablet_bootstrap.cc:492] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432: No bootstrap required, opened a new log
I20260924 16:12:15.705412 10167 ts_tablet_manager.cc:1397] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260924 16:12:15.705904 10167 raft_consensus.cc:348] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.706038 10167 raft_consensus.cc:374] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.706082 10167 raft_consensus.cc:729] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254a0a166d2f4261b2fc92503c87e432, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.706219 10167 consensus_queue.cc:260] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.706573 10167 ts_tablet_manager.cc:1428] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
W20260924 16:12:15.711274 10139 tablet.cc:2369] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:12:15.732963 9999 tablet.cc:2369] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:15.740952 10174 raft_consensus.cc:482] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:15.741106 10174 raft_consensus.cc:504] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.742264 10174 leader_election.cc:288] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66:45527), 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67:43113)
I20260924 16:12:15.746933 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.742632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:44647 (local address 127.1.253.66:45527)
0924 16:12:15.742753 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.742758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.742974 (+ 216us) server_negotiation.cc:407] Connection header received
0924 16:12:15.743133 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.743137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.743197 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.743276 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.743543 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.744171 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.745623 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.746337 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.746488 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.746548 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.746755 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.746827 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 16:12:15.747105 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.742880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:34447 (local address 127.1.253.67:43113)
0924 16:12:15.743010 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.743015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.743031 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:12:15.743411 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.743415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.743474 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.743541 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.743823 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.744592 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.745891 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.746374 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.746524 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.746578 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.746904 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.746981 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:12:15.747190 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.742425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:45527 (local address 127.1.253.65:44647)
0924 16:12:15.742896 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.742911 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.743031 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.743294 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.743296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.743310 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.743416 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.743421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.744281 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.744286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.745483 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.745494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.746482 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.746485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.746599 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.746624 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.746721 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.746801 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":25,"spinlock_wait_cycles":896,"thread_start_us":101,"threads_started":1}
I20260924 16:12:15.747521 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.742509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:43113 (local address 127.1.253.65:34447)
0924 16:12:15.742896 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.742911 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.743022 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.743580 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.743582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.743601 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.743709 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.743714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.744713 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.744717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.745779 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.745786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.746643 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.746648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.746756 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.746781 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.747293 (+ 512us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.747369 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260924 16:12:15.747825 9952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b555ec56dd274f4895203a77d7b0bef2" candidate_uuid: "061877b5d65e403d898b7d580cf6f9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c74b33aa9840b3b6df3c4381f6b454" is_pre_election: true
I20260924 16:12:15.747978 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b555ec56dd274f4895203a77d7b0bef2" candidate_uuid: "061877b5d65e403d898b7d580cf6f9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254a0a166d2f4261b2fc92503c87e432" is_pre_election: true
I20260924 16:12:15.748126 10092 raft_consensus.cc:2454] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 061877b5d65e403d898b7d580cf6f9d1 in term 0.
I20260924 16:12:15.748109 9952 raft_consensus.cc:2454] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 061877b5d65e403d898b7d580cf6f9d1 in term 0.
I20260924 16:12:15.748571 9745 leader_election.cc:302] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 061877b5d65e403d898b7d580cf6f9d1, 254a0a166d2f4261b2fc92503c87e432, 74c74b33aa9840b3b6df3c4381f6b454; no voters:
I20260924 16:12:15.748832 10174 raft_consensus.cc:2781] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:15.748883 10174 raft_consensus.cc:482] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:15.748919 10174 raft_consensus.cc:3037] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.749979 10174 raft_consensus.cc:504] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.750397 10174 leader_election.cc:288] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [CANDIDATE]: Term 1 election: Requested vote from peers 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66:45527), 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67:43113)
I20260924 16:12:15.750713 9952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b555ec56dd274f4895203a77d7b0bef2" candidate_uuid: "061877b5d65e403d898b7d580cf6f9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c74b33aa9840b3b6df3c4381f6b454"
I20260924 16:12:15.750828 9952 raft_consensus.cc:3037] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.752004 9952 raft_consensus.cc:2454] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 061877b5d65e403d898b7d580cf6f9d1 in term 1.
I20260924 16:12:15.752247 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b555ec56dd274f4895203a77d7b0bef2" candidate_uuid: "061877b5d65e403d898b7d580cf6f9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254a0a166d2f4261b2fc92503c87e432"
I20260924 16:12:15.752365 10092 raft_consensus.cc:3037] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.753487 10092 raft_consensus.cc:2454] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 061877b5d65e403d898b7d580cf6f9d1 in term 1.
I20260924 16:12:15.753815 9744 leader_election.cc:302] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 061877b5d65e403d898b7d580cf6f9d1, 254a0a166d2f4261b2fc92503c87e432; no voters:
I20260924 16:12:15.753989 10174 raft_consensus.cc:2781] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:15.754196 10174 raft_consensus.cc:686] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 LEADER]: Becoming Leader. State: Replica: 061877b5d65e403d898b7d580cf6f9d1, State: Running, Role: LEADER
I20260924 16:12:15.754487 10174 consensus_queue.cc:237] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
W20260924 16:12:15.754752 9859 tablet.cc:2369] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:15.756979 9654 catalog_manager.cc:5495] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65). New cstate: current_term: 1 leader_uuid: "061877b5d65e403d898b7d580cf6f9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } health_report { overall_health: UNKNOWN } } }
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266334, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.126@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
I20260924 16:12:15.797593 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.788110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.1.253.126:33135)
0924 16:12:15.788261 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.788266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.788427 (+ 161us) server_negotiation.cc:407] Connection header received
0924 16:12:15.788529 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.788534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.788589 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.788700 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.789326 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.790122 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.790965 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.791268 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.791371 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.794097 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.794100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.794102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.794123 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.795606 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.796153 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.796156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.796157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.796228 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.796578 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.796581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.796582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.796755 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.796837 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.797297 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.797462 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":67}
I20260924 16:12:15.797598 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.788030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:33135 (local address 127.0.0.1:60158)
0924 16:12:15.788317 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.788340 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.788445 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.788731 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.788733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.789106 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:15.789199 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.789204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.790289 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.790293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.790823 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.790831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.791408 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.791410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.791574 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.792194 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:15.792205 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:15.793929 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:15.795725 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.795736 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.795743 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.796044 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.796323 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.796326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.796329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.796437 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.796896 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:15.796903 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:15.797068 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.797208 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.797384 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":108,"threads_started":1}
I20260924 16:12:15.802081 9654 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:60158:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:12:15.803082 9654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:15.808514 9792 tablet_service.cc:1467] Processing CreateTablet for tablet 9caa128b82734b528f110d425fac8aa6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e706f51fafd4f1b97afbd5963492fad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:12:15.808810 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 9caa128b82734b528f110d425fac8aa6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e706f51fafd4f1b97afbd5963492fad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:12:15.808969 9792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9caa128b82734b528f110d425fac8aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.808836 9932 tablet_service.cc:1467] Processing CreateTablet for tablet 9caa128b82734b528f110d425fac8aa6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e706f51fafd4f1b97afbd5963492fad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:12:15.809160 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9caa128b82734b528f110d425fac8aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.809239 9932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9caa128b82734b528f110d425fac8aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.812222 10166 tablet_bootstrap.cc:492] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454: Bootstrap starting.
I20260924 16:12:15.812296 10170 tablet_bootstrap.cc:492] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1: Bootstrap starting.
I20260924 16:12:15.812831 10167 tablet_bootstrap.cc:492] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: Bootstrap starting.
I20260924 16:12:15.813251 10166 tablet_bootstrap.cc:654] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.813273 10170 tablet_bootstrap.cc:654] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.813812 10167 tablet_bootstrap.cc:654] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.814936 10170 tablet_bootstrap.cc:492] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1: No bootstrap required, opened a new log
I20260924 16:12:15.815026 10170 ts_tablet_manager.cc:1397] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:15.815203 10167 tablet_bootstrap.cc:492] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: No bootstrap required, opened a new log
I20260924 16:12:15.815268 10167 ts_tablet_manager.cc:1397] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:15.815548 10166 tablet_bootstrap.cc:492] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454: No bootstrap required, opened a new log
I20260924 16:12:15.815483 10170 raft_consensus.cc:348] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.815590 10170 raft_consensus.cc:374] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.815624 10166 ts_tablet_manager.cc:1397] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:15.815622 10170 raft_consensus.cc:729] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061877b5d65e403d898b7d580cf6f9d1, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.815665 10167 raft_consensus.cc:348] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.815774 10167 raft_consensus.cc:374] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.815747 10170 consensus_queue.cc:260] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.815804 10167 raft_consensus.cc:729] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254a0a166d2f4261b2fc92503c87e432, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.815917 10167 consensus_queue.cc:260] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.816032 10170 ts_tablet_manager.cc:1428] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:15.816051 10166 raft_consensus.cc:348] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.816154 10167 ts_tablet_manager.cc:1428] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:15.816159 10166 raft_consensus.cc:374] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.816192 10166 raft_consensus.cc:729] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c74b33aa9840b3b6df3c4381f6b454, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.816315 10166 consensus_queue.cc:260] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.816339 10174 tablet_replica.cc:439] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } } }
I20260924 16:12:15.816473 10174 tablet_replica.cc:442] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:15.816572 10166 ts_tablet_manager.cc:1428] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:15.816759 10173 tablet_replica.cc:439] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } } }
I20260924 16:12:15.816884 10173 tablet_replica.cc:442] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:15.817240 10172 tablet_replica.cc:439] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } } }
I20260924 16:12:15.817374 10172 tablet_replica.cc:442] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:15.820638 10172 raft_consensus.cc:482] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:15.820734 10172 raft_consensus.cc:504] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.821838 10172 leader_election.cc:288] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359), 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66:45527)
I20260924 16:12:15.826296 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.822312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:45527 (local address 127.1.253.67:40123)
0924 16:12:15.822664 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.822679 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.822788 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.823150 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.823153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.823171 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.823267 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.823273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.824152 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.824157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.825153 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.825163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.825824 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.825827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.825928 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.825951 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.826041 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.826117 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"spinlock_wait_cycles":22656,"thread_start_us":136,"threads_started":1}
I20260924 16:12:15.826309 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.822384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:40123 (local address 127.1.253.66:45527)
0924 16:12:15.822524 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.822528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.822874 (+ 346us) server_negotiation.cc:407] Connection header received
0924 16:12:15.822919 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.822922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.822964 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.823050 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.823405 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.824048 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.825298 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.825703 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.825833 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.825889 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.826059 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.826146 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:12:15.826834 9952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9caa128b82734b528f110d425fac8aa6" candidate_uuid: "254a0a166d2f4261b2fc92503c87e432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c74b33aa9840b3b6df3c4381f6b454" is_pre_election: true
I20260924 16:12:15.826974 9952 raft_consensus.cc:2454] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 254a0a166d2f4261b2fc92503c87e432 in term 0.
I20260924 16:12:15.827349 10025 leader_election.cc:302] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 254a0a166d2f4261b2fc92503c87e432, 74c74b33aa9840b3b6df3c4381f6b454; no voters:
I20260924 16:12:15.827663 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.822112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:48053 (local address 127.1.253.65:37359)
0924 16:12:15.822266 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.822272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.822764 (+ 492us) server_negotiation.cc:407] Connection header received
0924 16:12:15.822877 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.822881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.822930 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.823015 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.823742 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.824444 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.825993 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.826679 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.826794 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.826829 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.827451 (+ 622us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.827539 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:12:15.827648 10172 raft_consensus.cc:2781] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:15.827646 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.822040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:37359 (local address 127.1.253.67:48053)
0924 16:12:15.822664 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.822679 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.822788 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.823042 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.823045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.823060 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.823608 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.823616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.824597 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.824601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.825826 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.825838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.827185 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.827189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.827297 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.827324 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.827414 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.827498 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":4,"spinlock_wait_cycles":8832}
I20260924 16:12:15.827729 10172 raft_consensus.cc:482] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:15.827772 10172 raft_consensus.cc:3037] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.829067 10172 raft_consensus.cc:504] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.829406 10172 leader_election.cc:288] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [CANDIDATE]: Term 1 election: Requested vote from peers 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359), 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66:45527)
I20260924 16:12:15.829903 9952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9caa128b82734b528f110d425fac8aa6" candidate_uuid: "254a0a166d2f4261b2fc92503c87e432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c74b33aa9840b3b6df3c4381f6b454"
I20260924 16:12:15.830036 9952 raft_consensus.cc:3037] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.830003 9812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9caa128b82734b528f110d425fac8aa6" candidate_uuid: "254a0a166d2f4261b2fc92503c87e432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "061877b5d65e403d898b7d580cf6f9d1" is_pre_election: true
I20260924 16:12:15.830128 9812 raft_consensus.cc:2454] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 254a0a166d2f4261b2fc92503c87e432 in term 0.
I20260924 16:12:15.830397 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9caa128b82734b528f110d425fac8aa6" candidate_uuid: "254a0a166d2f4261b2fc92503c87e432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "061877b5d65e403d898b7d580cf6f9d1"
I20260924 16:12:15.830516 9811 raft_consensus.cc:3037] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.831274 9952 raft_consensus.cc:2454] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 254a0a166d2f4261b2fc92503c87e432 in term 1.
I20260924 16:12:15.831593 10025 leader_election.cc:302] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 254a0a166d2f4261b2fc92503c87e432, 74c74b33aa9840b3b6df3c4381f6b454; no voters:
I20260924 16:12:15.831790 10172 raft_consensus.cc:2781] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:15.831794 9811 raft_consensus.cc:2454] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 254a0a166d2f4261b2fc92503c87e432 in term 1.
I20260924 16:12:15.832008 10172 raft_consensus.cc:686] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [term 1 LEADER]: Becoming Leader. State: Replica: 254a0a166d2f4261b2fc92503c87e432, State: Running, Role: LEADER
I20260924 16:12:15.832257 10172 consensus_queue.cc:237] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:15.834542 10208 tablet_replica.cc:439] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: TxnStatusTablet state changed. Reason: New leader 254a0a166d2f4261b2fc92503c87e432. Latest consensus state: current_term: 1 leader_uuid: "254a0a166d2f4261b2fc92503c87e432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } } }
I20260924 16:12:15.834657 10208 tablet_replica.cc:442] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:12:15.834955 10210 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:12:15.835106 10210 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:12:15.835283 9653 catalog_manager.cc:5495] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67). New cstate: current_term: 1 leader_uuid: "254a0a166d2f4261b2fc92503c87e432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } health_report { overall_health: HEALTHY } } }
I20260924 16:12:15.851759 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.846953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.1.253.67:43113)
0924 16:12:15.847067 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.847072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.847210 (+ 138us) server_negotiation.cc:407] Connection header received
0924 16:12:15.847349 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.847354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.847438 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.847605 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:15.848028 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.848847 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.850050 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.850404 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.850519 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.850816 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:15.851315 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:15.851472 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.851557 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.851616 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:12:15.851773 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.846875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:43113 (local address 127.0.0.1:59556)
0924 16:12:15.847100 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.847114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.847217 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.847764 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.847768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.847779 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:12:15.847895 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.847901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.848968 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.848972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.849928 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.849939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.850521 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.850525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.850619 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.850674 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:12:15.851448 (+ 774us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:12:15.851465 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.851548 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.851630 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260924 16:12:15.854907 9811 raft_consensus.cc:1261] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Refusing update from remote peer 254a0a166d2f4261b2fc92503c87e432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:15.855043 9952 raft_consensus.cc:1261] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Refusing update from remote peer 254a0a166d2f4261b2fc92503c87e432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:15.855401 10208 consensus_queue.cc:1035] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:15.855587 10172 consensus_queue.cc:1035] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:15.856968 10174 tablet_replica.cc:439] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1: TxnStatusTablet state changed. Reason: New leader 254a0a166d2f4261b2fc92503c87e432. Latest consensus state: current_term: 1 leader_uuid: "254a0a166d2f4261b2fc92503c87e432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } } }
I20260924 16:12:15.857087 10174 tablet_replica.cc:442] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:15.857256 10173 tablet_replica.cc:439] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454: TxnStatusTablet state changed. Reason: New leader 254a0a166d2f4261b2fc92503c87e432. Latest consensus state: current_term: 1 leader_uuid: "254a0a166d2f4261b2fc92503c87e432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } } }
I20260924 16:12:15.857388 10173 tablet_replica.cc:442] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:15.859586 10172 tablet_replica.cc:439] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "254a0a166d2f4261b2fc92503c87e432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } } }
I20260924 16:12:15.859700 10172 tablet_replica.cc:442] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:12:15.860507 10208 tablet_replica.cc:439] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "254a0a166d2f4261b2fc92503c87e432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } } }
I20260924 16:12:15.860615 10208 tablet_replica.cc:442] T 9caa128b82734b528f110d425fac8aa6 P 254a0a166d2f4261b2fc92503c87e432: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:12:15.860610 10173 tablet_replica.cc:439] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "254a0a166d2f4261b2fc92503c87e432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } } }
I20260924 16:12:15.860705 10173 tablet_replica.cc:442] T 9caa128b82734b528f110d425fac8aa6 P 74c74b33aa9840b3b6df3c4381f6b454: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:15.860786 10179 mvcc.cc:205] Tried to move back new op lower bound from 7332930911656034304 to 7332930911571513344. Current Snapshot: MvccSnapshot[applied={T|T < 7332930911656034304 or (T in {7332930911656034304})}]
I20260924 16:12:15.862020 10174 tablet_replica.cc:439] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "254a0a166d2f4261b2fc92503c87e432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } } }
I20260924 16:12:15.862134 10174 tablet_replica.cc:442] T 9caa128b82734b528f110d425fac8aa6 P 061877b5d65e403d898b7d580cf6f9d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:15.869338 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.865550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.1.253.65:37359)
0924 16:12:15.865693 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.865699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.865716 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:12:15.865899 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.865902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.865966 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.866074 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:15.866332 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.866959 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.867797 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.868051 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.868162 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.868436 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:15.868896 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:15.869033 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.869105 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.869180 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:12:15.881114 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.877004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:36837 (local address 127.1.253.67:43113)
0924 16:12:15.877150 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.877155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.877704 (+ 549us) server_negotiation.cc:407] Connection header received
0924 16:12:15.877817 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.877819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.877863 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.877925 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:15.878202 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.878751 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.879977 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.880186 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.880277 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.880625 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:15.880695 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:15.880748 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.880965 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.881016 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:12:15.881125 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.877328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:43113 (local address 127.1.253.65:36837)
0924 16:12:15.877604 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.877618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.877721 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.877962 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.877965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.877974 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:12:15.878099 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.878105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.878858 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.878862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.879847 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.879859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.880316 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.880320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.880430 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.880482 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:12:15.880778 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:12:15.880785 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.880871 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.880950 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260924 16:12:15.888285 10092 raft_consensus.cc:1261] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Refusing update from remote peer 061877b5d65e403d898b7d580cf6f9d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:15.888286 9952 raft_consensus.cc:1261] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Refusing update from remote peer 061877b5d65e403d898b7d580cf6f9d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:15.888803 10196 consensus_queue.cc:1035] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:15.889006 10174 consensus_queue.cc:1035] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:15.912451 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.908656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:37359 (local address 127.1.253.67:38361)
0924 16:12:15.908829 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.908848 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.908929 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.909173 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.909176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.909184 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:12:15.909283 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.909287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.910267 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.910271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.911237 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.911247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.911810 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.911813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.911882 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.911928 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:12:15.912221 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:12:15.912226 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.912275 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.912341 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 16:12:15.912472 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.908724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:38361 (local address 127.1.253.65:37359)
0924 16:12:15.908833 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.908838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.908889 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:12:15.909025 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.909028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.909078 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.909157 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:15.909379 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.910150 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.911353 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.911704 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.911808 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.912053 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:15.912137 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:15.912207 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.912285 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.912341 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:12:15.936570 9655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:15.936769 9655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:15.939172 9655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7: Sending DeleteTablet for 3 replicas of tablet b555ec56dd274f4895203a77d7b0bef2
I20260924 16:12:15.939917 9932 tablet_service.cc:1514] Processing DeleteTablet for tablet b555ec56dd274f4895203a77d7b0bef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:15 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:44526
I20260924 16:12:15.939924 9792 tablet_service.cc:1514] Processing DeleteTablet for tablet b555ec56dd274f4895203a77d7b0bef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:15 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:33728
I20260924 16:12:15.940121 10190 tablet_replica.cc:330] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454: stopping tablet replica
I20260924 16:12:15.940129 10189 tablet_replica.cc:330] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1: stopping tablet replica
I20260924 16:12:15.940258 10189 raft_consensus.cc:2229] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:15.940284 10190 raft_consensus.cc:2229] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:15.940418 10190 raft_consensus.cc:2258] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:15.940570 10189 raft_consensus.cc:2258] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:15.941159 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet b555ec56dd274f4895203a77d7b0bef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:15 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:59540
I20260924 16:12:15.941298 10190 ts_tablet_manager.cc:1905] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:15.941608 10189 ts_tablet_manager.cc:1905] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:15.944218 10190 ts_tablet_manager.cc:1918] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:15.944218 10189 ts_tablet_manager.cc:1918] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:15.944288 10190 log.cc:1196] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/b555ec56dd274f4895203a77d7b0bef2
I20260924 16:12:15.944301 10189 log.cc:1196] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/wal/wals/b555ec56dd274f4895203a77d7b0bef2
I20260924 16:12:15.944557 10190 ts_tablet_manager.cc:1939] T b555ec56dd274f4895203a77d7b0bef2 P 74c74b33aa9840b3b6df3c4381f6b454: Deleting consensus metadata
I20260924 16:12:15.944561 10189 ts_tablet_manager.cc:1939] T b555ec56dd274f4895203a77d7b0bef2 P 061877b5d65e403d898b7d580cf6f9d1: Deleting consensus metadata
I20260924 16:12:15.944805 10188 tablet_replica.cc:330] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432: stopping tablet replica
I20260924 16:12:15.944919 10188 raft_consensus.cc:2229] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:15.945029 10188 raft_consensus.cc:2258] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:15.945358 9641 catalog_manager.cc:4841] TS 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359): tablet b555ec56dd274f4895203a77d7b0bef2 (table test-table [id=4b0f972ecfe34bd78f94ca2cc4de4d3d]) successfully deleted
I20260924 16:12:15.946122 9641 catalog_manager.cc:4841] TS 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66:45527): tablet b555ec56dd274f4895203a77d7b0bef2 (table test-table [id=4b0f972ecfe34bd78f94ca2cc4de4d3d]) successfully deleted
I20260924 16:12:15.946382 10188 ts_tablet_manager.cc:1905] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:15.948752 10188 ts_tablet_manager.cc:1918] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:15.948846 10188 log.cc:1196] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/b555ec56dd274f4895203a77d7b0bef2
I20260924 16:12:15.949082 10188 ts_tablet_manager.cc:1939] T b555ec56dd274f4895203a77d7b0bef2 P 254a0a166d2f4261b2fc92503c87e432: Deleting consensus metadata
I20260924 16:12:15.949734 9640 catalog_manager.cc:4841] TS 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67:43113): tablet b555ec56dd274f4895203a77d7b0bef2 (table test-table [id=4b0f972ecfe34bd78f94ca2cc4de4d3d]) successfully deleted
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:12:15.975939 9704 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:15.969704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.1.253.126:33135)
0924 16:12:15.969840 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.969845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.970121 (+ 276us) server_negotiation.cc:407] Connection header received
0924 16:12:15.970335 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.970340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.970404 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.970495 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.971193 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.971705 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.972545 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.972765 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.972872 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.975831 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60160: BlockingRecv error: recv got EOF from 127.0.0.1:60160 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 16:12:15 dist-test-slave-6wvm krb5kdc[9610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM
I20260924 16:12:15.987084 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.977849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.1.253.126:33135)
0924 16:12:15.977958 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.977962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.978392 (+ 430us) server_negotiation.cc:407] Connection header received
0924 16:12:15.978515 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.978518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.978566 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.978649 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.979243 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.979781 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.980631 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.980851 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.980958 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.983383 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.983387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.983390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.983416 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.985201 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.985698 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.985701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.985703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.985791 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.986070 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.986073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.986075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.986382 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.986523 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.986771 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.986881 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:12:15.989807 9655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:15.990351 9655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:15.995782 9792 tablet_service.cc:1467] Processing CreateTablet for tablet f9ba68f449844325889b2b0421c84fe5 (DEFAULT_TABLE table=test-table [id=8e626abb72c7478992a84cfda8502bc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.995903 10072 tablet_service.cc:1467] Processing CreateTablet for tablet f9ba68f449844325889b2b0421c84fe5 (DEFAULT_TABLE table=test-table [id=8e626abb72c7478992a84cfda8502bc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.996163 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ba68f449844325889b2b0421c84fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.996165 9792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ba68f449844325889b2b0421c84fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.998724 9932 tablet_service.cc:1467] Processing CreateTablet for tablet f9ba68f449844325889b2b0421c84fe5 (DEFAULT_TABLE table=test-table [id=8e626abb72c7478992a84cfda8502bc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.998986 10167 tablet_bootstrap.cc:492] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432: Bootstrap starting.
I20260924 16:12:15.999044 9932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ba68f449844325889b2b0421c84fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:16.000008 10167 tablet_bootstrap.cc:654] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:16.001559 10166 tablet_bootstrap.cc:492] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454: Bootstrap starting.
I20260924 16:12:16.002503 10166 tablet_bootstrap.cc:654] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:16.003101 10170 tablet_bootstrap.cc:492] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1: Bootstrap starting.
I20260924 16:12:16.004074 10170 tablet_bootstrap.cc:654] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:16.006668 10170 tablet_bootstrap.cc:492] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1: No bootstrap required, opened a new log
I20260924 16:12:16.006747 10170 ts_tablet_manager.cc:1397] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 16:12:16.007184 10170 raft_consensus.cc:348] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.007297 10170 raft_consensus.cc:374] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:16.007329 10170 raft_consensus.cc:729] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061877b5d65e403d898b7d580cf6f9d1, State: Initialized, Role: FOLLOWER
I20260924 16:12:16.007460 10170 consensus_queue.cc:260] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.007753 10170 ts_tablet_manager.cc:1428] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:16.011744 10167 tablet_bootstrap.cc:492] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432: No bootstrap required, opened a new log
I20260924 16:12:16.011844 10167 ts_tablet_manager.cc:1397] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20260924 16:12:16.012274 10167 raft_consensus.cc:348] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.012372 10167 raft_consensus.cc:374] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:16.012398 10167 raft_consensus.cc:729] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254a0a166d2f4261b2fc92503c87e432, State: Initialized, Role: FOLLOWER
I20260924 16:12:16.012506 10167 consensus_queue.cc:260] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.012765 10167 ts_tablet_manager.cc:1428] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 16:12:16.019026 10166 tablet_bootstrap.cc:492] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454: No bootstrap required, opened a new log
I20260924 16:12:16.019115 10166 ts_tablet_manager.cc:1397] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20260924 16:12:16.019488 10166 raft_consensus.cc:348] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.019591 10166 raft_consensus.cc:374] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:16.019618 10166 raft_consensus.cc:729] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c74b33aa9840b3b6df3c4381f6b454, State: Initialized, Role: FOLLOWER
I20260924 16:12:16.019727 10166 consensus_queue.cc:260] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.019965 10166 ts_tablet_manager.cc:1428] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:16.224150 10173 raft_consensus.cc:482] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:16.224321 10173 raft_consensus.cc:504] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.224735 10173 leader_election.cc:288] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359), 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67:43113)
I20260924 16:12:16.225275 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ba68f449844325889b2b0421c84fe5" candidate_uuid: "74c74b33aa9840b3b6df3c4381f6b454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254a0a166d2f4261b2fc92503c87e432" is_pre_election: true
I20260924 16:12:16.225282 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ba68f449844325889b2b0421c84fe5" candidate_uuid: "74c74b33aa9840b3b6df3c4381f6b454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "061877b5d65e403d898b7d580cf6f9d1" is_pre_election: true
I20260924 16:12:16.225445 10092 raft_consensus.cc:2454] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c74b33aa9840b3b6df3c4381f6b454 in term 0.
I20260924 16:12:16.225461 9811 raft_consensus.cc:2454] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c74b33aa9840b3b6df3c4381f6b454 in term 0.
I20260924 16:12:16.225778 9884 leader_election.cc:302] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 254a0a166d2f4261b2fc92503c87e432, 74c74b33aa9840b3b6df3c4381f6b454; no voters:
I20260924 16:12:16.225953 10173 raft_consensus.cc:2781] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:16.226016 10173 raft_consensus.cc:482] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:16.226069 10173 raft_consensus.cc:3037] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:16.227156 10173 raft_consensus.cc:504] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.227535 10173 leader_election.cc:288] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [CANDIDATE]: Term 1 election: Requested vote from peers 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359), 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67:43113)
I20260924 16:12:16.227921 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ba68f449844325889b2b0421c84fe5" candidate_uuid: "74c74b33aa9840b3b6df3c4381f6b454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254a0a166d2f4261b2fc92503c87e432"
I20260924 16:12:16.227945 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ba68f449844325889b2b0421c84fe5" candidate_uuid: "74c74b33aa9840b3b6df3c4381f6b454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "061877b5d65e403d898b7d580cf6f9d1"
I20260924 16:12:16.228053 10092 raft_consensus.cc:3037] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:16.228084 9811 raft_consensus.cc:3037] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:16.228973 10092 raft_consensus.cc:2454] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c74b33aa9840b3b6df3c4381f6b454 in term 1.
I20260924 16:12:16.229000 9811 raft_consensus.cc:2454] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c74b33aa9840b3b6df3c4381f6b454 in term 1.
I20260924 16:12:16.229314 9884 leader_election.cc:302] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 061877b5d65e403d898b7d580cf6f9d1, 254a0a166d2f4261b2fc92503c87e432, 74c74b33aa9840b3b6df3c4381f6b454; no voters:
I20260924 16:12:16.229452 10173 raft_consensus.cc:2781] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:16.229609 10173 raft_consensus.cc:686] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 LEADER]: Becoming Leader. State: Replica: 74c74b33aa9840b3b6df3c4381f6b454, State: Running, Role: LEADER
I20260924 16:12:16.229763 10173 consensus_queue.cc:237] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.231477 9655 catalog_manager.cc:5495] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66). New cstate: current_term: 1 leader_uuid: "74c74b33aa9840b3b6df3c4381f6b454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } health_report { overall_health: UNKNOWN } } }
I20260924 16:12:16.249812 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.246339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.1.253.66:45527)
0924 16:12:16.246455 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:16.246460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:16.246567 (+ 107us) server_negotiation.cc:407] Connection header received
0924 16:12:16.246751 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:16.246757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:16.246817 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:16.246896 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:16.247194 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.247837 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.248712 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.248961 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.249079 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.249362 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:16.249444 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:16.249530 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:16.249629 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 16:12:16.249681 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 16:12:16.251791 9811 raft_consensus.cc:1261] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Refusing update from remote peer 74c74b33aa9840b3b6df3c4381f6b454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:16.251793 10092 raft_consensus.cc:1261] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Refusing update from remote peer 74c74b33aa9840b3b6df3c4381f6b454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:16.252257 10173 consensus_queue.cc:1035] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:16.252630 10173 consensus_queue.cc:1035] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:16.259617 9655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:16.259754 9655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:16.261684 9655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7: Sending DeleteTablet for 3 replicas of tablet f9ba68f449844325889b2b0421c84fe5
I20260924 16:12:16.262308 9792 tablet_service.cc:1514] Processing DeleteTablet for tablet f9ba68f449844325889b2b0421c84fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:16 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:33728
I20260924 16:12:16.262399 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet f9ba68f449844325889b2b0421c84fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:16 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:59540
I20260924 16:12:16.262533 9932 tablet_service.cc:1514] Processing DeleteTablet for tablet f9ba68f449844325889b2b0421c84fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:16 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:44526
I20260924 16:12:16.262727 10189 tablet_replica.cc:330] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1: stopping tablet replica
I20260924 16:12:16.262866 10189 raft_consensus.cc:2229] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:16.262890 10188 tablet_replica.cc:330] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432: stopping tablet replica
I20260924 16:12:16.262997 10188 raft_consensus.cc:2229] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:16.263025 10189 raft_consensus.cc:2258] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:16.263139 10188 raft_consensus.cc:2258] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:16.263199 10190 tablet_replica.cc:330] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454: stopping tablet replica
I20260924 16:12:16.263322 10190 raft_consensus.cc:2229] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:16.263532 10190 raft_consensus.cc:2258] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:16.263691 9655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:16.264156 9655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:16.264245 10188 ts_tablet_manager.cc:1905] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:16.264320 10190 ts_tablet_manager.cc:1905] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:16.264379 10189 ts_tablet_manager.cc:1905] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:16.266810 10188 ts_tablet_manager.cc:1918] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:16.266821 10190 ts_tablet_manager.cc:1918] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:16.266889 10188 log.cc:1196] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/f9ba68f449844325889b2b0421c84fe5
I20260924 16:12:16.266892 10190 log.cc:1196] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/f9ba68f449844325889b2b0421c84fe5
I20260924 16:12:16.267079 10188 ts_tablet_manager.cc:1939] T f9ba68f449844325889b2b0421c84fe5 P 254a0a166d2f4261b2fc92503c87e432: Deleting consensus metadata
I20260924 16:12:16.267155 10190 ts_tablet_manager.cc:1939] T f9ba68f449844325889b2b0421c84fe5 P 74c74b33aa9840b3b6df3c4381f6b454: Deleting consensus metadata
I20260924 16:12:16.267656 10189 ts_tablet_manager.cc:1918] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:16.267709 10189 log.cc:1196] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-0/wal/wals/f9ba68f449844325889b2b0421c84fe5
I20260924 16:12:16.267951 10189 ts_tablet_manager.cc:1939] T f9ba68f449844325889b2b0421c84fe5 P 061877b5d65e403d898b7d580cf6f9d1: Deleting consensus metadata
I20260924 16:12:16.267922 9641 catalog_manager.cc:4841] TS 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66:45527): tablet f9ba68f449844325889b2b0421c84fe5 (table test-table [id=8e626abb72c7478992a84cfda8502bc3]) successfully deleted
I20260924 16:12:16.268038 9640 catalog_manager.cc:4841] TS 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67:43113): tablet f9ba68f449844325889b2b0421c84fe5 (table test-table [id=8e626abb72c7478992a84cfda8502bc3]) successfully deleted
I20260924 16:12:16.268558 9641 catalog_manager.cc:4841] TS 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359): tablet f9ba68f449844325889b2b0421c84fe5 (table test-table [id=8e626abb72c7478992a84cfda8502bc3]) successfully deleted
I20260924 16:12:16.271425 9792 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5ab277903946f0b13a792caf40fc98 (DEFAULT_TABLE table=test-table [id=427a2b3c700d4914a81dff6c6c4d0491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:16.271456 9932 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5ab277903946f0b13a792caf40fc98 (DEFAULT_TABLE table=test-table [id=427a2b3c700d4914a81dff6c6c4d0491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:16.271598 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 6b5ab277903946f0b13a792caf40fc98 (DEFAULT_TABLE table=test-table [id=427a2b3c700d4914a81dff6c6c4d0491]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:16.271684 9932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5ab277903946f0b13a792caf40fc98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:16.271690 9792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5ab277903946f0b13a792caf40fc98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:16.271824 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b5ab277903946f0b13a792caf40fc98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:16.273885 10170 tablet_bootstrap.cc:492] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1: Bootstrap starting.
I20260924 16:12:16.273942 10166 tablet_bootstrap.cc:492] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454: Bootstrap starting.
I20260924 16:12:16.273978 10167 tablet_bootstrap.cc:492] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432: Bootstrap starting.
I20260924 16:12:16.274724 10166 tablet_bootstrap.cc:654] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:16.274770 10170 tablet_bootstrap.cc:654] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:16.274770 10167 tablet_bootstrap.cc:654] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:16.275780 10166 tablet_bootstrap.cc:492] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454: No bootstrap required, opened a new log
I20260924 16:12:16.275848 10166 ts_tablet_manager.cc:1397] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:16.275842 10167 tablet_bootstrap.cc:492] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432: No bootstrap required, opened a new log
I20260924 16:12:16.275920 10167 ts_tablet_manager.cc:1397] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:16.276250 10166 raft_consensus.cc:348] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.276335 10166 raft_consensus.cc:374] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:16.276356 10166 raft_consensus.cc:729] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c74b33aa9840b3b6df3c4381f6b454, State: Initialized, Role: FOLLOWER
I20260924 16:12:16.276300 10167 raft_consensus.cc:348] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.276417 10167 raft_consensus.cc:374] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:16.276454 10167 raft_consensus.cc:729] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254a0a166d2f4261b2fc92503c87e432, State: Initialized, Role: FOLLOWER
I20260924 16:12:16.276491 10166 consensus_queue.cc:260] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.276615 10167 consensus_queue.cc:260] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.276798 10166 ts_tablet_manager.cc:1428] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:16.276880 10167 ts_tablet_manager.cc:1428] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 16:12:16.277258 10170 tablet_bootstrap.cc:492] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1: No bootstrap required, opened a new log
I20260924 16:12:16.277354 10170 ts_tablet_manager.cc:1397] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 16:12:16.277699 10170 raft_consensus.cc:348] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.277809 10170 raft_consensus.cc:374] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:16.277853 10170 raft_consensus.cc:729] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061877b5d65e403d898b7d580cf6f9d1, State: Initialized, Role: FOLLOWER
I20260924 16:12:16.277984 10170 consensus_queue.cc:260] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.278287 10170 ts_tablet_manager.cc:1428] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 16:12:16.335021 10208 raft_consensus.cc:482] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:16.335171 10208 raft_consensus.cc:504] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.335657 10208 leader_election.cc:288] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359), 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66:45527)
I20260924 16:12:16.336208 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5ab277903946f0b13a792caf40fc98" candidate_uuid: "254a0a166d2f4261b2fc92503c87e432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "061877b5d65e403d898b7d580cf6f9d1" is_pre_election: true
I20260924 16:12:16.336292 9950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5ab277903946f0b13a792caf40fc98" candidate_uuid: "254a0a166d2f4261b2fc92503c87e432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c74b33aa9840b3b6df3c4381f6b454" is_pre_election: true
I20260924 16:12:16.336387 9811 raft_consensus.cc:2454] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 254a0a166d2f4261b2fc92503c87e432 in term 0.
I20260924 16:12:16.336458 9950 raft_consensus.cc:2454] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 254a0a166d2f4261b2fc92503c87e432 in term 0.
I20260924 16:12:16.336771 10025 leader_election.cc:302] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 254a0a166d2f4261b2fc92503c87e432, 74c74b33aa9840b3b6df3c4381f6b454; no voters:
I20260924 16:12:16.337030 10208 raft_consensus.cc:2781] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:16.337097 10208 raft_consensus.cc:482] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:16.337136 10208 raft_consensus.cc:3037] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:16.338099 10208 raft_consensus.cc:504] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.338426 10208 leader_election.cc:288] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [CANDIDATE]: Term 1 election: Requested vote from peers 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359), 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66:45527)
I20260924 16:12:16.338798 9811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5ab277903946f0b13a792caf40fc98" candidate_uuid: "254a0a166d2f4261b2fc92503c87e432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "061877b5d65e403d898b7d580cf6f9d1"
I20260924 16:12:16.338798 9950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b5ab277903946f0b13a792caf40fc98" candidate_uuid: "254a0a166d2f4261b2fc92503c87e432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c74b33aa9840b3b6df3c4381f6b454"
I20260924 16:12:16.338915 9811 raft_consensus.cc:3037] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:16.338915 9950 raft_consensus.cc:3037] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:16.340051 9950 raft_consensus.cc:2454] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 254a0a166d2f4261b2fc92503c87e432 in term 1.
I20260924 16:12:16.340051 9811 raft_consensus.cc:2454] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 254a0a166d2f4261b2fc92503c87e432 in term 1.
I20260924 16:12:16.340354 10025 leader_election.cc:302] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 254a0a166d2f4261b2fc92503c87e432, 74c74b33aa9840b3b6df3c4381f6b454; no voters:
I20260924 16:12:16.340526 10208 raft_consensus.cc:2781] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:16.340619 10208 raft_consensus.cc:686] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 1 LEADER]: Becoming Leader. State: Replica: 254a0a166d2f4261b2fc92503c87e432, State: Running, Role: LEADER
I20260924 16:12:16.340765 10208 consensus_queue.cc:237] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } }
I20260924 16:12:16.342475 9654 catalog_manager.cc:5495] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67). New cstate: current_term: 1 leader_uuid: "254a0a166d2f4261b2fc92503c87e432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "254a0a166d2f4261b2fc92503c87e432" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 43113 } health_report { overall_health: HEALTHY } } }
I20260924 16:12:16.364131 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.361352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.1.253.67:43113)
0924 16:12:16.361471 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:16.361474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:16.361486 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:12:16.361578 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:16.361580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:16.361619 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:16.361677 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:16.361900 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.362455 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.363234 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.363438 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.363515 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.363715 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:16.363779 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:16.363831 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:16.363908 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 16:12:16.363949 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 16:12:16.372574 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.369821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:56533 (local address 127.1.253.67:43113)
0924 16:12:16.369959 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:16.369962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:16.369974 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:12:16.370060 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:16.370062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:16.370099 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:16.370158 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:16.370472 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.370918 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.371686 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.371872 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.371943 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.372152 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:16.372209 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:16.372262 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:16.372407 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 16:12:16.372465 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":45}
I20260924 16:12:16.372578 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.369751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:43113 (local address 127.1.253.67:56533)
0924 16:12:16.369889 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:16.369910 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:16.369980 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:16.370182 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:16.370184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:16.370190 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:12:16.370360 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:16.370370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.371039 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.371042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:16.371580 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:16.371588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.371963 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.371965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:16.372033 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.372056 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:12:16.372308 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:12:16.372315 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:12:16.372383 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 16:12:16.372476 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":15}
I20260924 16:12:16.377849 9950 raft_consensus.cc:1261] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Refusing update from remote peer 254a0a166d2f4261b2fc92503c87e432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:16.377875 9811 raft_consensus.cc:1261] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Refusing update from remote peer 254a0a166d2f4261b2fc92503c87e432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:16.378280 10208 consensus_queue.cc:1035] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c74b33aa9840b3b6df3c4381f6b454" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 45527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:16.378412 10218 consensus_queue.cc:1035] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "061877b5d65e403d898b7d580cf6f9d1" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:16.411311 9653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:16.411489 9653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:16.413559 9653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64d3b7abf07c46efab55e10d94ad40b7: Sending DeleteTablet for 3 replicas of tablet 6b5ab277903946f0b13a792caf40fc98
I20260924 16:12:16.414134 9792 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5ab277903946f0b13a792caf40fc98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:16 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:33728
I20260924 16:12:16.414222 9932 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5ab277903946f0b13a792caf40fc98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:16 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:44526
I20260924 16:12:16.414377 10189 tablet_replica.cc:330] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1: stopping tablet replica
I20260924 16:12:16.414444 10190 tablet_replica.cc:330] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454: stopping tablet replica
I20260924 16:12:16.414507 10189 raft_consensus.cc:2229] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:16.414499 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b5ab277903946f0b13a792caf40fc98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:16 UTC) from {username='oryx', principal='oryx/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:59540
I20260924 16:12:16.414582 10190 raft_consensus.cc:2229] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:16.414624 10189 raft_consensus.cc:2258] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:16.414664 10188 tablet_replica.cc:330] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432: stopping tablet replica
I20260924 16:12:16.414691 10190 raft_consensus.cc:2258] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:16.414776 10188 raft_consensus.cc:2229] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:16.415100 10188 raft_consensus.cc:2258] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:16.415501 10190 ts_tablet_manager.cc:1905] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:16.415858 10188 ts_tablet_manager.cc:1905] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:16.418145 10190 ts_tablet_manager.cc:1918] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:16.418259 10190 log.cc:1196] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/6b5ab277903946f0b13a792caf40fc98
I20260924 16:12:16.418325 10188 ts_tablet_manager.cc:1918] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:16.418401 10188 log.cc:1196] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/6b5ab277903946f0b13a792caf40fc98
I20260924 16:12:16.418589 10190 ts_tablet_manager.cc:1939] T 6b5ab277903946f0b13a792caf40fc98 P 74c74b33aa9840b3b6df3c4381f6b454: Deleting consensus metadata
I20260924 16:12:16.418664 10188 ts_tablet_manager.cc:1939] T 6b5ab277903946f0b13a792caf40fc98 P 254a0a166d2f4261b2fc92503c87e432: Deleting consensus metadata
I20260924 16:12:16.419327 9640 catalog_manager.cc:4841] TS 254a0a166d2f4261b2fc92503c87e432 (127.1.253.67:43113): tablet 6b5ab277903946f0b13a792caf40fc98 (table test-table [id=427a2b3c700d4914a81dff6c6c4d0491]) successfully deleted
I20260924 16:12:16.419466 9641 catalog_manager.cc:4841] TS 74c74b33aa9840b3b6df3c4381f6b454 (127.1.253.66:45527): tablet 6b5ab277903946f0b13a792caf40fc98 (table test-table [id=427a2b3c700d4914a81dff6c6c4d0491]) successfully deleted
I20260924 16:12:16.419754 10189 ts_tablet_manager.cc:1905] T 6b5ab277903946f0b13a792caf40fc98 P 061877b5d65e403d898b7d580cf6f9d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:16.419777 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 9727
W20260924 16:12:16.429885 9641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:37359 (error 108)
I20260924 16:12:16.430130 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 9867
I20260924 16:12:16.432128 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.431635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:37359 (local address 127.0.0.1:49530)
0924 16:12:16.431902 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:16.431987 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":64,"threads_started":1}
W20260924 16:12:16.432483 9641 catalog_manager.cc:4584] TS 061877b5d65e403d898b7d580cf6f9d1 (127.1.253.65:37359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b5ab277903946f0b13a792caf40fc98: Network error: Client connection negotiation failed: client connection to 127.1.253.65:37359: connect: Connection refused (error 111)
I20260924 16:12:16.438364 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 10007
I20260924 16:12:16.445525 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 9623
2026-09-24T16:12:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3087 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:16 dist-test-slave-6wvm krb5kdc[10258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:16 dist-test-slave-6wvm krb5kdc[10258](info): set up 2 sockets
Sep 24 16:12:16 dist-test-slave-6wvm krb5kdc[10258](info): commencing operation
krb5kdc: starting...
W20260924 16:12:17.502720 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:12:17 dist-test-slave-6wvm krb5kdc[10258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:12:17.628927 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:43323
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:43323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:17.742734 10271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:17.743014 10271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:17.743055 10271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:17.747023 10271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:17.747084 10271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:17.747104 10271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:17.747121 10271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:17.747136 10271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:17.752389 10271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:43323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:43323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:17.753490 10271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:17.754629 10271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:17.763531 10276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:17.763597 10277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:17.763499 10279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:17.763748 10271 server_base.cc:1034] running on GCE node
I20260924 16:12:17.764410 10271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:17.765614 10271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:17.766803 10271 hybrid_clock.cc:648] HybridClock initialized: now 1790266337766789 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T16:12:17Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1330 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:17 dist-test-slave-6wvm krb5kdc[10284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:17 dist-test-slave-6wvm krb5kdc[10284](info): set up 2 sockets
Sep 24 16:12:17 dist-test-slave-6wvm krb5kdc[10284](info): commencing operation
krb5kdc: starting...
W20260924 16:12:18.818992 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:12:18 dist-test-slave-6wvm krb5kdc[10284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:12:18.944793 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:44947
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:44947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:19.057723 10297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:19.058038 10297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:19.058111 10297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:19.062134 10297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:19.062240 10297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:19.062309 10297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:19.062367 10297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:19.062408 10297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:19.067790 10297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:44947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:19.069018 10297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:19.070187 10297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:19.079243 10302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:19.079321 10297 server_base.cc:1034] running on GCE node
W20260924 16:12:19.079265 10303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:19.079267 10305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:19.080133 10297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:19.081278 10297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:19.082441 10297 hybrid_clock.cc:648] HybridClock initialized: now 1790266339082416 us; error 38 us; skew 500 ppm
Sep 24 16:12:19 dist-test-slave-6wvm krb5kdc[10284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266339, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:19.089264 10297 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:12:19.090294 10297 webserver.cc:460] Webserver started at http://127.1.253.126:34703/ using document root <none> and password file <none>
I20260924 16:12:19.090893 10297 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:19.090984 10297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:19.091228 10297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:19.093324 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "ad4624ac76c647a095a6cd94015e6e24"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "8787605dda916ea1c396f787476b6497"
server_key_iv: "6f54e128a8e224ed48173d4602b0ff10"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.093910 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "ad4624ac76c647a095a6cd94015e6e24"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "8787605dda916ea1c396f787476b6497"
server_key_iv: "6f54e128a8e224ed48173d4602b0ff10"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.097837 10297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 16:12:19.100476 10312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.101869 10297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:19.101979 10297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "ad4624ac76c647a095a6cd94015e6e24"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "8787605dda916ea1c396f787476b6497"
server_key_iv: "6f54e128a8e224ed48173d4602b0ff10"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.102080 10297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:19.123713 10297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:19.126576 10297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:19.126729 10297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:19.133360 10297 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:44947
I20260924 16:12:19.133440 10364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:44947 every 8 connection(s)
I20260924 16:12:19.134418 10297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:19.137221 10365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:19.141894 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 10297
I20260924 16:12:19.142005 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:19.142375 2037 external_mini_cluster.cc:1428] Setting key adad4a77f0bb448be9bcddad6d414ebd
I20260924 16:12:19.143194 10365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24: Bootstrap starting.
I20260924 16:12:19.145924 10365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:19.146828 10365 log.cc:824] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:19.148887 10365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24: No bootstrap required, opened a new log
Sep 24 16:12:19 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:19.152700 10365 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4624ac76c647a095a6cd94015e6e24" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44947 } }
I20260924 16:12:19.153062 10365 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:19.153156 10365 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4624ac76c647a095a6cd94015e6e24, State: Initialized, Role: FOLLOWER
I20260924 16:12:19.153743 10365 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4624ac76c647a095a6cd94015e6e24" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44947 } }
I20260924 16:12:19.153911 10365 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:19.154008 10365 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:19.154117 10365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:19.155076 10365 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4624ac76c647a095a6cd94015e6e24" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44947 } }
I20260924 16:12:19.155450 10365 leader_election.cc:302] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad4624ac76c647a095a6cd94015e6e24; no voters:
I20260924 16:12:19.155827 10365 leader_election.cc:288] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:19.155907 10370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:19.156157 10370 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 1 LEADER]: Becoming Leader. State: Replica: ad4624ac76c647a095a6cd94015e6e24, State: Running, Role: LEADER
I20260924 16:12:19.156525 10370 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4624ac76c647a095a6cd94015e6e24" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44947 } }
I20260924 16:12:19.157338 10365 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:19.158006 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.143930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.1.253.126:44947)
0924 16:12:19.144402 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:19.144424 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:19.144468 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:12:19.145290 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:19.145315 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:19.145738 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:19.146112 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:19.146851 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.148204 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.149100 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.149573 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.149746 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.152391 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:19.152396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:19.152410 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:19.152439 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:19.154489 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.154942 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.154948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.154954 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.155081 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.155388 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.155393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.155395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.155875 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:19.156093 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:19.156862 (+ 769us) server_negotiation.cc:299] Negotiation successful
0924 16:12:19.157105 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":116,"threads_started":1}
I20260924 16:12:19.158602 10372 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad4624ac76c647a095a6cd94015e6e24. Latest consensus state: current_term: 1 leader_uuid: "ad4624ac76c647a095a6cd94015e6e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4624ac76c647a095a6cd94015e6e24" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44947 } } }
I20260924 16:12:19.158730 10372 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:19.159027 10371 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad4624ac76c647a095a6cd94015e6e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4624ac76c647a095a6cd94015e6e24" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44947 } } }
I20260924 16:12:19.159111 10371 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:19.159142 10379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:19.162451 10379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:19.167886 10379 catalog_manager.cc:1347] Generated new cluster ID: d440f2a2b5774bbdade2973ffb45e030
I20260924 16:12:19.167958 10379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:19.176625 10379 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:19.177826 10379 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:19.192193 10379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24: Generated new TSK 0
I20260924 16:12:19.193042 10379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:12:19.230777 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:44947
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:19.369964 10391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:19.370255 10391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:19.370312 10391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:19.374464 10391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:19.374521 10391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:19.374586 10391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:12:19.379976 10391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:44947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:19.381039 10391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:19.382059 10391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:12:19.392167 10391 server_base.cc:1034] running on GCE node
W20260924 16:12:19.392102 10397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:19.392130 10399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:19.392096 10396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:19.392870 10391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:19.393466 10391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:19.394620 10391 hybrid_clock.cc:648] HybridClock initialized: now 1790266339394606 us; error 25 us; skew 500 ppm
Sep 24 16:12:19 dist-test-slave-6wvm krb5kdc[10284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266339, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:19.397290 10391 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:12:19.398214 10391 webserver.cc:460] Webserver started at http://127.1.253.65:41201/ using document root <none> and password file <none>
I20260924 16:12:19.398829 10391 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:19.398921 10391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:19.399152 10391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:19.401206 10391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "44ea525c783f4188aecde4c84693d7d0"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "148d23361feb4d7fde8c210a922d165d"
server_key_iv: "efefe5af45426631f36cd33274cf624d"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.401787 10391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "44ea525c783f4188aecde4c84693d7d0"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "148d23361feb4d7fde8c210a922d165d"
server_key_iv: "efefe5af45426631f36cd33274cf624d"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.405915 10391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:12:19.409039 10406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.410404 10391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:12:19.410542 10391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "44ea525c783f4188aecde4c84693d7d0"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "148d23361feb4d7fde8c210a922d165d"
server_key_iv: "efefe5af45426631f36cd33274cf624d"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.410673 10391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:19.434895 10391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:19.437963 10391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:19.438186 10391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:19.438843 10391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:19.439714 10391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:19.439786 10391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.439860 10391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:19.439904 10391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.448617 10391 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:33209
I20260924 16:12:19.448628 10519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:33209 every 8 connection(s)
I20260924 16:12:19.449608 10391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:19.458931 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 10391
I20260924 16:12:19.459075 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:19.459416 2037 external_mini_cluster.cc:1428] Setting key 3ea7091c35c16755f4a60b20b8073c77
Sep 24 16:12:19 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266339, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:19.464306 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.451590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:42783 (local address 127.1.253.126:44947)
0924 16:12:19.451744 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:19.451748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:19.452810 (+ 1062us) server_negotiation.cc:407] Connection header received
0924 16:12:19.453762 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:19.453766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:19.453830 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:19.453941 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:19.455242 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.455784 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.456962 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.457229 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.457360 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.460186 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:19.460189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:19.460191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:19.460230 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:19.461756 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.462430 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.462432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.462434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.462513 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.462981 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.462988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.462992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.463278 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:19.463388 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:19.463928 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 16:12:19.464093 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:12:19.465101 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.451943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:42783)
0924 16:12:19.452643 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:19.452688 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:19.453530 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:19.454137 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:19.454144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:19.454582 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:19.455081 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.455095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.455923 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.455932 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.456824 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.456834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.457416 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.457419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.457576 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.458168 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:19.458186 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:19.459995 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:19.461886 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.461895 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.461916 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.462328 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.462606 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.462612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.462618 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.462831 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.463393 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:19.463399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:19.463653 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 16:12:19.463822 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 16:12:19.464103 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":122,"threads_started":1}
I20260924 16:12:19.466409 10520 heartbeater.cc:347] Connected to a master server at 127.1.253.126:44947
I20260924 16:12:19.466696 10520 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:19.467229 10520 heartbeater.cc:511] Master 127.1.253.126:44947 requested a full tablet report, sending...
I20260924 16:12:19.468899 10326 ts_manager.cc:194] Registered new tserver with Master: 44ea525c783f4188aecde4c84693d7d0 (127.1.253.65:33209)
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:12:19.470163 10326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:42783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:12:19.489563 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:44947
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:19.606343 10525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:19.606690 10525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:19.606734 10525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:19.610824 10525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:19.610880 10525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:19.610939 10525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:12:19.616268 10525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:44947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:19.617322 10525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:19.618378 10525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:19.628005 10533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:19.628078 10525 server_base.cc:1034] running on GCE node
W20260924 16:12:19.627995 10530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:19.627991 10531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:19.628598 10525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:19.629318 10525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:19.630496 10525 hybrid_clock.cc:648] HybridClock initialized: now 1790266339630480 us; error 26 us; skew 500 ppm
Sep 24 16:12:19 dist-test-slave-6wvm krb5kdc[10284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266339, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:19.633285 10525 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:12:19.634145 10525 webserver.cc:460] Webserver started at http://127.1.253.66:45129/ using document root <none> and password file <none>
I20260924 16:12:19.634765 10525 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:19.634859 10525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:19.635102 10525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:19.637271 10525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "0085501a048344428dcbb64b36995b46"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "840aad9ef8cef54297bbe98c59ae0eb0"
server_key_iv: "e996ba4b13d6da5c604356f32b123661"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.637871 10525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "0085501a048344428dcbb64b36995b46"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "840aad9ef8cef54297bbe98c59ae0eb0"
server_key_iv: "e996ba4b13d6da5c604356f32b123661"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.641856 10525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:12:19.644461 10540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.645792 10525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:12:19.645929 10525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "0085501a048344428dcbb64b36995b46"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "840aad9ef8cef54297bbe98c59ae0eb0"
server_key_iv: "e996ba4b13d6da5c604356f32b123661"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.646042 10525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:19.664417 10525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:19.667915 10525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:19.668128 10525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:19.668751 10525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:19.669616 10525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:19.669687 10525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.669760 10525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:19.669803 10525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.678876 10525 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:45629
I20260924 16:12:19.678894 10653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:45629 every 8 connection(s)
I20260924 16:12:19.679795 10525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:19.686944 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 10525
I20260924 16:12:19.687067 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:19.687402 2037 external_mini_cluster.cc:1428] Setting key ae2087b4d2e4df68bd91c3a67384249a
Sep 24 16:12:19 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266339, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:19.694304 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.681617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:53417 (local address 127.1.253.126:44947)
0924 16:12:19.681791 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:19.681796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:19.682912 (+ 1116us) server_negotiation.cc:407] Connection header received
0924 16:12:19.683767 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:19.683770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:19.683819 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:19.683886 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:19.685042 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.685599 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.686795 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.687160 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.687293 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.690193 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:19.690196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:19.690198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:19.690224 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:19.692007 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.692530 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.692537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.692542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.692636 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.692965 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.692971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.692976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.693243 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:19.693349 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:19.693873 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 16:12:19.694018 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:12:19.695156 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.682051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:53417)
0924 16:12:19.682760 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:19.682803 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:19.683551 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:19.684057 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:19.684063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:19.684410 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:19.684900 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.684914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.685711 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.685713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.686641 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.686651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.687338 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.687341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.687534 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.688183 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:19.688203 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:19.690014 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:19.692141 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.692145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.692158 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.692426 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.692733 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.692735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.692736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.692870 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.693347 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:19.693351 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:19.693588 (+ 237us) client_negotiation.cc:769] Sending connection context
0924 16:12:19.693788 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 16:12:19.694170 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":166,"threads_started":1}
I20260924 16:12:19.696398 10654 heartbeater.cc:347] Connected to a master server at 127.1.253.126:44947
I20260924 16:12:19.696660 10654 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:19.697167 10654 heartbeater.cc:511] Master 127.1.253.126:44947 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:12:19.698302 10326 ts_manager.cc:194] Registered new tserver with Master: 0085501a048344428dcbb64b36995b46 (127.1.253.66:45629)
I20260924 16:12:19.698884 10326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:53417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:12:19.717635 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:44947
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:19.831115 10659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:19.831365 10659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:19.831431 10659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:19.835450 10659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:19.835511 10659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:19.835601 10659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:12:19.840883 10659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:44947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:19.841933 10659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:19.843039 10659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:19.852986 10667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:19.853024 10664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:19.853163 10659 server_base.cc:1034] running on GCE node
W20260924 16:12:19.853015 10665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:19.853631 10659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:19.854373 10659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:19.855531 10659 hybrid_clock.cc:648] HybridClock initialized: now 1790266339855498 us; error 44 us; skew 500 ppm
Sep 24 16:12:19 dist-test-slave-6wvm krb5kdc[10284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266339, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:19.858497 10659 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:12:19.859356 10659 webserver.cc:460] Webserver started at http://127.1.253.67:44375/ using document root <none> and password file <none>
I20260924 16:12:19.859937 10659 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:19.860033 10659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:19.860296 10659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:19.862453 10659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "be75c689c7af4af08e3fc8b61b2c5a4f"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "56cfff32bd10cac76118b838bd0f9b57"
server_key_iv: "3dfc4948b283b6d496bbd79900780b69"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.863025 10659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "be75c689c7af4af08e3fc8b61b2c5a4f"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "56cfff32bd10cac76118b838bd0f9b57"
server_key_iv: "3dfc4948b283b6d496bbd79900780b69"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.866830 10659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:12:19.869235 10674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.870553 10659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:19.870687 10659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "be75c689c7af4af08e3fc8b61b2c5a4f"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "56cfff32bd10cac76118b838bd0f9b57"
server_key_iv: "3dfc4948b283b6d496bbd79900780b69"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.870797 10659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:19.897071 10659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:19.900823 10659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:19.901052 10659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:19.901711 10659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:19.902626 10659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:19.902699 10659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.902776 10659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:19.902829 10659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.912405 10659 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:45807
I20260924 16:12:19.912428 10787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:45807 every 8 connection(s)
I20260924 16:12:19.913349 10659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:19.914544 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 10659
I20260924 16:12:19.914697 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:19.915041 2037 external_mini_cluster.cc:1428] Setting key 7ce5d518973ae0ed4b3292129725b17d
Sep 24 16:12:19 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266339, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:19.927119 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.915605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:46735 (local address 127.1.253.126:44947)
0924 16:12:19.915815 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:19.915818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:19.917111 (+ 1293us) server_negotiation.cc:407] Connection header received
0924 16:12:19.918071 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:19.918074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:19.918124 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:19.918220 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:19.919447 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.919958 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.921066 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.921278 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.921351 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.923878 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:19.923881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:19.923883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:19.923909 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:19.925159 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.925707 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.925710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.925711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.925781 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.926118 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.926120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.926121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.926347 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:19.926424 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:19.926905 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 16:12:19.926997 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260924 16:12:19.927992 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.916093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:46735)
0924 16:12:19.916962 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:19.917018 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:19.917863 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:19.918448 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:19.918454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:19.918827 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:19.919320 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.919334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.920068 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.920071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.920959 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.920968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.921407 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.921410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.921581 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.922125 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:19.922142 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:19.923710 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:19.925301 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.925306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.925320 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.925615 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.925868 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.925871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.925872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.926036 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.926445 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:19.926451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:19.926693 (+ 242us) client_negotiation.cc:769] Sending connection context
0924 16:12:19.926864 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 16:12:19.927126 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":175,"threads_started":1}
I20260924 16:12:19.929200 10788 heartbeater.cc:347] Connected to a master server at 127.1.253.126:44947
I20260924 16:12:19.929476 10788 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:19.929976 10788 heartbeater.cc:511] Master 127.1.253.126:44947 requested a full tablet report, sending...
I20260924 16:12:19.931146 10328 ts_manager.cc:194] Registered new tserver with Master: be75c689c7af4af08e3fc8b61b2c5a4f (127.1.253.67:45807)
I20260924 16:12:19.931803 10328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:46735
I20260924 16:12:19.940763 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab.
Entry for principal oryx/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab.
WARNING: no policy specified for oryx/127.1.253.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab.
Entry for principal oryx/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab.
WARNING: no policy specified for oryx/127.1.253.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.253.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab.
Entry for principal oryx/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab.
I20260924 16:12:20.021260 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 10391
I20260924 16:12:20.028157 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 10525
I20260924 16:12:20.034217 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 10659
I20260924 16:12:20.040268 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 10297
I20260924 16:12:20.046990 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:44947
--webserver_interface=127.1.253.126
--webserver_port=34703
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:44947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:20.163486 10797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:20.163849 10797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:20.163937 10797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:20.168103 10797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:20.168205 10797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:20.168243 10797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:20.168282 10797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:20.168315 10797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:20.173707 10797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:44947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=34703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:20.174888 10797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:20.176021 10797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:20.184692 10803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:20.184799 10797 server_base.cc:1034] running on GCE node
W20260924 16:12:20.184692 10805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:20.184732 10802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:20.185585 10797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:20.186717 10797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:20.187958 10797 hybrid_clock.cc:648] HybridClock initialized: now 1790266340187938 us; error 29 us; skew 500 ppm
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266340, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:20.190837 10797 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:12:20.191829 10797 webserver.cc:460] Webserver started at http://127.1.253.126:34703/ using document root <none> and password file <none>
I20260924 16:12:20.192418 10797 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:20.192546 10797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:20.196455 10797 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:20.198843 10812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:20.200186 10797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:20.200340 10797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "ad4624ac76c647a095a6cd94015e6e24"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "8787605dda916ea1c396f787476b6497"
server_key_iv: "6f54e128a8e224ed48173d4602b0ff10"
server_key_version: "encryptionkey@0"
I20260924 16:12:20.200901 10797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:20.243703 10797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:20.246887 10797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:20.247115 10797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:20.254516 10797 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:44947
I20260924 16:12:20.254535 10864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:44947 every 8 connection(s)
I20260924 16:12:20.255625 10797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:20.260035 10865 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:12:20.261803 10865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24: Bootstrap starting.
I20260924 16:12:20.264169 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 10797
I20260924 16:12:20.264652 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:33209
--local_ip_for_outbound_sockets=127.1.253.65
--tserver_master_addrs=127.1.253.126:44947
--webserver_port=41201
--webserver_interface=127.1.253.65
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.253.65
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:12:20.283535 10865 log.cc:824] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:20.292507 10865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:12:20.292943 10865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24: Bootstrap complete.
I20260924 16:12:20.298511 10865 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4624ac76c647a095a6cd94015e6e24" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44947 } }
I20260924 16:12:20.299021 10865 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4624ac76c647a095a6cd94015e6e24, State: Initialized, Role: FOLLOWER
I20260924 16:12:20.299623 10865 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4624ac76c647a095a6cd94015e6e24" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44947 } }
I20260924 16:12:20.299779 10865 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:20.299841 10865 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:20.299973 10865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:12:20.302654 10865 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4624ac76c647a095a6cd94015e6e24" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44947 } }
I20260924 16:12:20.303025 10865 leader_election.cc:302] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad4624ac76c647a095a6cd94015e6e24; no voters:
I20260924 16:12:20.303381 10865 leader_election.cc:288] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:12:20.303535 10869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:12:20.303817 10869 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [term 2 LEADER]: Becoming Leader. State: Replica: ad4624ac76c647a095a6cd94015e6e24, State: Running, Role: LEADER
I20260924 16:12:20.304184 10869 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4624ac76c647a095a6cd94015e6e24" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44947 } }
I20260924 16:12:20.304754 10865 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:20.305701 10871 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad4624ac76c647a095a6cd94015e6e24. Latest consensus state: current_term: 2 leader_uuid: "ad4624ac76c647a095a6cd94015e6e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4624ac76c647a095a6cd94015e6e24" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44947 } } }
I20260924 16:12:20.305820 10871 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:20.306138 10870 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ad4624ac76c647a095a6cd94015e6e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4624ac76c647a095a6cd94015e6e24" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44947 } } }
I20260924 16:12:20.306262 10870 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:20.306964 10878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:20.310479 10878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:20.311143 10878 catalog_manager.cc:1259] Loaded cluster ID: d440f2a2b5774bbdade2973ffb45e030
I20260924 16:12:20.311203 10878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:20.313586 10878 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:20.314260 10878 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad4624ac76c647a095a6cd94015e6e24: Loaded TSK: 0
I20260924 16:12:20.315477 10878 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:12:20.394794 10867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:20.395074 10867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:20.395159 10867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:20.399328 10867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:20.399415 10867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:20.399515 10867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:12:20.405148 10867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.65.keytab
--principal=oryx/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:33209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=41201
--tserver_master_addrs=127.1.253.126:44947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:20.406476 10867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:20.407486 10867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:20.418962 10894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:20.418990 10892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:20.419122 10867 server_base.cc:1034] running on GCE node
W20260924 16:12:20.418958 10891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:20.419842 10867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:20.420506 10867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:20.421705 10867 hybrid_clock.cc:648] HybridClock initialized: now 1790266340421674 us; error 48 us; skew 500 ppm
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266340, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:20.424657 10867 init.cc:377] Logged in from keytab as oryx/127.1.253.65@KRBTEST.COM (short username oryx)
I20260924 16:12:20.425655 10867 webserver.cc:460] Webserver started at http://127.1.253.65:41201/ using document root <none> and password file <none>
I20260924 16:12:20.426218 10867 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:20.426354 10867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:20.430495 10867 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:20.432796 10901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:20.434180 10867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:20.434355 10867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "44ea525c783f4188aecde4c84693d7d0"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "148d23361feb4d7fde8c210a922d165d"
server_key_iv: "efefe5af45426631f36cd33274cf624d"
server_key_version: "encryptionkey@0"
I20260924 16:12:20.434957 10867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:20.452937 10867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:20.456414 10867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:20.456632 10867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:20.457258 10867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:20.458113 10867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:20.458161 10867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:20.458202 10867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:20.458290 10867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:20.467715 10867 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:33209
I20260924 16:12:20.467732 11014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:33209 every 8 connection(s)
I20260924 16:12:20.468778 10867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:20.472925 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 10867
I20260924 16:12:20.473445 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:45629
--local_ip_for_outbound_sockets=127.1.253.66
--tserver_master_addrs=127.1.253.126:44947
--webserver_port=45129
--webserver_interface=127.1.253.66
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.253.66
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:12:20.483532 11017 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:20.471030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:41749 (local address 127.1.253.126:44947)
0924 16:12:20.471499 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.471515 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.472002 (+ 487us) server_negotiation.cc:407] Connection header received
0924 16:12:20.473133 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.473147 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.473550 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.473928 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:20.475093 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.476082 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.477906 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.478521 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.478655 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.482763 (+ 4108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:41749: BlockingRecv error: recv got EOF from 127.1.253.65:41749 (error 108)
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":143,"threads_started":1}
I20260924 16:12:20.483544 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.471112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:41749)
0924 16:12:20.471829 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.471882 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.472684 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.474016 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.474023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.474398 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:20.474895 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.474909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.476236 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.476242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.477765 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.477781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.478707 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.478715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.478960 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.479626 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:20.479654 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:20.482526 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":141,"threads_started":1}
W20260924 16:12:20.484076 11015 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:44947 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.253.126:44947: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:20.489809 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.484599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:45119)
0924 16:12:20.484727 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.484741 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.484827 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.485069 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.485072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.485241 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:20.485332 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.485337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.486084 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.486089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.486705 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.486713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.487160 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.487164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.487256 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.487766 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:20.487783 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:20.489649 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 16:12:20.489877 11017 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:20.484715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:45119 (local address 127.1.253.126:44947)
0924 16:12:20.484834 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.484838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.484854 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:12:20.484905 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.484908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.484959 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.485054 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:20.485450 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.485976 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.486826 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.487046 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.487157 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.489770 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:45119: BlockingRecv error: recv got EOF from 127.1.253.65:45119 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:20.495703 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.490414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:55413)
0924 16:12:20.490528 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.490540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.490629 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.491009 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.491012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.491128 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:20.491205 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.491210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.492275 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.492277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.492780 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.492787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.493336 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.493339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.493401 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.493882 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:20.493894 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:20.495595 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 16:12:20.495898 11017 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:20.490674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:55413 (local address 127.1.253.126:44947)
0924 16:12:20.490804 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.490808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.490822 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:12:20.490859 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.490862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.490903 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.490983 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:20.491312 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.492161 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.492887 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.493229 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.493329 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.495767 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:55413: BlockingRecv error: recv got EOF from 127.1.253.65:55413 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 16:12:20.495944 11015 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:12:20.593217 11019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:20.593474 11019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:20.593518 11019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:20.597445 11019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:20.597515 11019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:20.597570 11019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:12:20.602821 11019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.66.keytab
--principal=oryx/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:45629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=45129
--tserver_master_addrs=127.1.253.126:44947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:20.603858 11019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:20.605020 11019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:20.615370 11024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:20.615377 11025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:20.615489 11019 server_base.cc:1034] running on GCE node
W20260924 16:12:20.615713 11027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:20.616154 11019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:20.616844 11019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:20.618016 11019 hybrid_clock.cc:648] HybridClock initialized: now 1790266340617984 us; error 40 us; skew 500 ppm
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266340, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:20.620759 11019 init.cc:377] Logged in from keytab as oryx/127.1.253.66@KRBTEST.COM (short username oryx)
I20260924 16:12:20.621645 11019 webserver.cc:460] Webserver started at http://127.1.253.66:45129/ using document root <none> and password file <none>
I20260924 16:12:20.622211 11019 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:20.622370 11019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:20.626186 11019 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:20.628225 11034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:20.629374 11019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:20.629506 11019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "0085501a048344428dcbb64b36995b46"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "840aad9ef8cef54297bbe98c59ae0eb0"
server_key_iv: "e996ba4b13d6da5c604356f32b123661"
server_key_version: "encryptionkey@0"
I20260924 16:12:20.630045 11019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:20.651117 11019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:20.654296 11019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:20.654517 11019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:20.655179 11019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:20.656047 11019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:20.656121 11019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:20.656193 11019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:20.656246 11019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:20.665426 11019 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:45629
I20260924 16:12:20.665453 11147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:45629 every 8 connection(s)
I20260924 16:12:20.666514 11019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:20.671435 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 11019
I20260924 16:12:20.671952 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:45807
--local_ip_for_outbound_sockets=127.1.253.67
--tserver_master_addrs=127.1.253.126:44947
--webserver_port=44375
--webserver_interface=127.1.253.67
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.253.67
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:12:20.678617 11017 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:20.668404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:39133 (local address 127.1.253.126:44947)
0924 16:12:20.668559 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.668564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.669736 (+ 1172us) server_negotiation.cc:407] Connection header received
0924 16:12:20.670623 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.670626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.670684 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.670768 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:20.672375 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.672905 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.674277 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.674649 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.674791 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.678511 (+ 3720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:39133: BlockingRecv error: recv got EOF from 127.1.253.66:39133 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:12:20.679545 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.668883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:39133)
0924 16:12:20.669576 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.669619 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.670409 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.670947 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.670954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.671456 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:20.672210 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.672233 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.673066 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.673073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.674122 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.674132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.674810 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.674813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.674952 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.675563 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:20.675580 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:20.678394 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":118,"threads_started":1}
W20260924 16:12:20.680110 11148 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:44947 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.253.126:44947: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:20.686040 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.680548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:50047)
0924 16:12:20.680758 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.680770 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.680829 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.681073 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.681075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.681196 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:20.681276 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.681280 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.682417 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.682419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.682940 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.682948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.683589 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.683592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.683659 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.684160 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:20.684171 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:20.685939 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260924 16:12:20.686211 11017 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:20.680672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:50047 (local address 127.1.253.126:44947)
0924 16:12:20.680825 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.680832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.680851 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:12:20.680901 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.680906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.680961 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.681061 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:20.681390 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.682311 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.683058 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.683471 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.683609 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.686096 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:50047: BlockingRecv error: recv got EOF from 127.1.253.66:50047 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:20.691671 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.686738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:40749)
0924 16:12:20.686844 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.686858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.686943 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.687181 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.687184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.687327 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:20.687390 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.687394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.688108 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.688113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.688626 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.688633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.689068 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.689071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.689151 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.689649 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:20.689659 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:20.691532 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 16:12:20.691859 11148 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:12:20.691994 11017 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:20.686853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:40749 (local address 127.1.253.126:44947)
0924 16:12:20.686986 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.686992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.687005 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:12:20.687040 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.687042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.687085 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.687166 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:20.687524 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.688000 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.688741 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.688958 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.689067 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.691668 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:40749: BlockingRecv error: recv got EOF from 127.1.253.66:40749 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 16:12:20.788648 11151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:20.788973 11151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:20.789047 11151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:20.793006 11151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:20.793107 11151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:20.793207 11151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:12:20.798405 11151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/krb5kdc/oryx_127.1.253.67.keytab
--principal=oryx/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:45807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=44375
--tserver_master_addrs=127.1.253.126:44947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:20.799628 11151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:20.801039 11151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:20.811087 11157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:20.811110 11156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:20.811260 11151 server_base.cc:1034] running on GCE node
W20260924 16:12:20.811419 11159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:20.811918 11151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:20.812562 11151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:20.813975 11151 hybrid_clock.cc:648] HybridClock initialized: now 1790266340813948 us; error 48 us; skew 500 ppm
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266340, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:20.816888 11151 init.cc:377] Logged in from keytab as oryx/127.1.253.67@KRBTEST.COM (short username oryx)
I20260924 16:12:20.817797 11151 webserver.cc:460] Webserver started at http://127.1.253.67:44375/ using document root <none> and password file <none>
I20260924 16:12:20.818413 11151 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:20.818495 11151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:20.822477 11151 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:20.824667 11166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:20.825989 11151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:20.826144 11151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "be75c689c7af4af08e3fc8b61b2c5a4f"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-6wvm"
server_key: "56cfff32bd10cac76118b838bd0f9b57"
server_key_iv: "3dfc4948b283b6d496bbd79900780b69"
server_key_version: "encryptionkey@0"
I20260924 16:12:20.826781 11151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:20.842310 11151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:20.845423 11151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:20.845619 11151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:20.846362 11151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:20.847230 11151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:20.847277 11151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:20.847347 11151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:20.847378 11151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:20.856487 11151 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:45807
I20260924 16:12:20.856513 11279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:45807 every 8 connection(s)
I20260924 16:12:20.857571 11151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:20.859174 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 11151
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:20.870018 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.860304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.1.253.126:44947)
0924 16:12:20.860550 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.860576 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.860596 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:12:20.860653 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.860658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.860725 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.860839 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:20.861726 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.862593 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.863455 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.863785 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.863896 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.865427 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:20.865434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:20.865452 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:20.865485 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:20.867721 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:20.868203 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:20.868209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:20.868213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:20.868332 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:20.868625 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:20.868628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:20.868630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:20.869046 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:20.869267 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:20.869547 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 16:12:20.869835 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":176,"thread_start_us":100,"threads_started":1}
W20260924 16:12:20.870385 11017 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:20.859737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:35849 (local address 127.1.253.126:44947)
0924 16:12:20.859887 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.859892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.861267 (+ 1375us) server_negotiation.cc:407] Connection header received
0924 16:12:20.862418 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.862421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.862466 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.862533 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:20.863910 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.864630 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.866062 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.866357 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.866494 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.870276 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:35849: BlockingRecv error: recv got EOF from 127.1.253.67:35849 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":65}
I20260924 16:12:20.871115 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.860265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:35849)
0924 16:12:20.861110 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.861162 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.862153 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.862754 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.862766 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.863126 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:20.863746 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.863766 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.864767 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.864770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.865900 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.865911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.866581 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.866587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.866808 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.867307 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:20.867329 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:20.870094 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":156,"threads_started":1}
W20260924 16:12:20.871623 11280 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:44947 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.253.126:44947: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:20.877130 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.872185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:55113)
0924 16:12:20.872316 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.872328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.872391 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.872824 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.872828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.872989 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:20.873107 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.873113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.873812 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.873817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.874471 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.874480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.874921 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.874924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.875015 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.875421 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:20.875432 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:20.877042 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:12:20.877321 11283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:20.872388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:55113 (local address 127.1.253.126:44947)
0924 16:12:20.872647 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.872650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.872659 (+ 9us) server_negotiation.cc:407] Connection header received
0924 16:12:20.872683 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.872685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.872716 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.872805 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:20.873221 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.873703 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.874590 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.874806 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.874919 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.877200 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:55113: BlockingRecv error: recv got EOF from 127.1.253.67:55113 (error 108)
Metrics: {"server-negotiator.queue_time_us":201}
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:20 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:20.882216 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.877608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:43637)
0924 16:12:20.877727 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.877746 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.877806 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.878002 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.878004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.878119 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:20.878186 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.878190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.878898 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.878900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.879398 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.879405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.879853 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.879855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.879909 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.880380 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:20.880392 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:20.882074 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:12:20.882335 11283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:20.877699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:43637 (local address 127.1.253.126:44947)
0924 16:12:20.877830 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.877833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.877846 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:12:20.877874 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.877876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.877909 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.877974 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:20.878330 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.878807 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.879544 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.879748 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.879839 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.882184 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:43637: BlockingRecv error: recv got EOF from 127.1.253.67:43637 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 16:12:20.882509 11280 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 16:12:21 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:21 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:21.502719 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:21.496680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:50441)
0924 16:12:21.496965 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:21.496983 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:21.497090 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:21.497387 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:21.497394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:21.497622 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:21.497726 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.497732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.498563 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.498566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.499149 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.499158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.499632 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.499635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.499700 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.500236 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:21.500252 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:21.502506 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
W20260924 16:12:21.502823 11286 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:21.496779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:50441 (local address 127.1.253.126:44947)
0924 16:12:21.497103 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:21.497107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:21.497122 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:12:21.497162 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:21.497164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:21.497219 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:21.497302 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:21.497852 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.498423 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.499282 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.499507 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.499587 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.502669 (+ 3082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:50441: BlockingRecv error: recv got EOF from 127.1.253.65:50441 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":159,"threads_started":1}
Sep 24 16:12:21 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:21 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:21.698072 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:21.692669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:49147)
0924 16:12:21.692992 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:21.693009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:21.693141 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:21.693436 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:21.693438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:21.693639 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:21.693744 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.693750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.694502 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.694509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.695084 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.695091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.695508 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.695511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.695573 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.696119 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:21.696131 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:21.697939 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":87,"threads_started":1}
W20260924 16:12:21.698372 11286 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:21.692747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:49147 (local address 127.1.253.126:44947)
0924 16:12:21.692906 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:21.692913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:21.693128 (+ 215us) server_negotiation.cc:407] Connection header received
0924 16:12:21.693222 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:21.693225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:21.693297 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:21.693385 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:21.693853 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.694404 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.695195 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.695409 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.695493 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.698095 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:49147: BlockingRecv error: recv got EOF from 127.1.253.66:49147 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 16:12:21 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:21 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:21.889149 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:21.883279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:35841)
0924 16:12:21.883655 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:21.883674 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:21.883810 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:21.884092 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:21.884095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:21.884304 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:21.884411 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.884417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.885189 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.885193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.885791 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.885801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.886401 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.886406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.886482 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.887097 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:21.887110 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:21.889016 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
W20260924 16:12:21.889305 11286 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:21.883382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:35841 (local address 127.1.253.126:44947)
0924 16:12:21.883539 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:21.883544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:21.883786 (+ 242us) server_negotiation.cc:407] Connection header received
0924 16:12:21.883899 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:21.883901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:21.883951 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:21.884018 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:21.884526 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.885068 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.885903 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.886122 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.886203 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.889166 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:35841: BlockingRecv error: recv got EOF from 127.1.253.67:35841 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 16:12:22 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:22 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:22.509445 11289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:22.503706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:48693)
0924 16:12:22.504042 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:22.504059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:22.504180 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:22.504462 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:22.504492 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:22.504687 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:22.504793 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:22.504799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:22.505666 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:22.505670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:22.506364 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:22.506373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:22.506795 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:22.506797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:22.506857 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:22.507436 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:22.507448 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:22.509298 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":141,"threads_started":1}
W20260924 16:12:22.509565 11290 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:22.503788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:48693 (local address 127.1.253.126:44947)
0924 16:12:22.504037 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:22.504040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:22.504145 (+ 105us) server_negotiation.cc:407] Connection header received
0924 16:12:22.504266 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:22.504269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:22.504314 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:22.504432 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:22.504918 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:22.505546 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:22.506473 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:22.506689 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:22.506772 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:22.509434 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:48693: BlockingRecv error: recv got EOF from 127.1.253.65:48693 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
Sep 24 16:12:22 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:22 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:22.704941 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:22.698968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:36195)
0924 16:12:22.699282 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:22.699300 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:22.699436 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:22.699822 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:22.699827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:22.700081 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:22.700189 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:22.700196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:22.701027 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:22.701032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:22.701604 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:22.701613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:22.702100 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:22.702104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:22.702195 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:22.702744 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:22.702756 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:22.704700 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
W20260924 16:12:22.704953 11290 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:22.699076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:36195 (local address 127.1.253.126:44947)
0924 16:12:22.699233 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:22.699238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:22.699389 (+ 151us) server_negotiation.cc:407] Connection header received
0924 16:12:22.699602 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:22.699605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:22.699661 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:22.699757 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:22.700334 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:22.700900 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:22.701733 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:22.701967 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:22.702093 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:22.704847 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:36195: BlockingRecv error: recv got EOF from 127.1.253.66:36195 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 16:12:22 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:22 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:22.896066 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:22.890199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:56639)
0924 16:12:22.890583 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:22.890605 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:22.890756 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:22.891049 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:22.891052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:22.891271 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:22.891373 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:22.891379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:22.892165 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:22.892170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:22.892754 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:22.892762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:22.893231 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:22.893235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:22.893327 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:22.893877 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:22.893890 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:22.895860 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":112,"threads_started":1}
W20260924 16:12:22.896104 11290 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:22.890372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:56639 (local address 127.1.253.126:44947)
0924 16:12:22.890538 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:22.890543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:22.890686 (+ 143us) server_negotiation.cc:407] Connection header received
0924 16:12:22.890869 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:22.890874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:22.890930 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:22.891020 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:22.891507 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:22.892039 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:22.892880 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:22.893106 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:22.893228 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:22.895997 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:56639: BlockingRecv error: recv got EOF from 127.1.253.67:56639 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 16:12:23 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:23 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:23.516141 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:23.510399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:54941)
0924 16:12:23.510723 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:23.510736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:23.510843 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:23.511100 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:23.511102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:23.511283 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:23.511385 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:23.511390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.512129 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.512131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:23.512829 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:23.512841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.513417 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.513421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:23.513510 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:23.514130 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:23.514142 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:23.515958 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":90,"threads_started":1}
W20260924 16:12:23.516254 11294 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:23.510411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:54941 (local address 127.1.253.126:44947)
0924 16:12:23.510673 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:23.510677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:23.510828 (+ 151us) server_negotiation.cc:407] Connection header received
0924 16:12:23.510932 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:23.510934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:23.510984 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:23.511060 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:23.511492 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.512030 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.512963 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.513288 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.513400 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:23.516100 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:54941: BlockingRecv error: recv got EOF from 127.1.253.65:54941 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
Sep 24 16:12:23 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:23 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:23.711560 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:23.705931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:56817)
0924 16:12:23.706217 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:23.706230 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:23.706397 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:23.706681 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:23.706684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:23.706890 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:23.706997 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:23.707002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.707756 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.707759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:23.708395 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:23.708405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.708863 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.708866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:23.708933 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:23.709467 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:23.709480 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:23.711432 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":88,"threads_started":1}
W20260924 16:12:23.711750 11294 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:23.705943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:56817 (local address 127.1.253.126:44947)
0924 16:12:23.706118 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:23.706122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:23.706327 (+ 205us) server_negotiation.cc:407] Connection header received
0924 16:12:23.706501 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:23.706504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:23.706554 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:23.706637 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:23.707088 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.707631 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.708519 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.708757 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.708840 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:23.711598 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:56817: BlockingRecv error: recv got EOF from 127.1.253.66:56817 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 16:12:23 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:23 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:23.902597 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:23.897085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:47965)
0924 16:12:23.897368 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:23.897382 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:23.897484 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:23.897716 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:23.897718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:23.897898 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:23.898008 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:23.898014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.898807 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.898809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:23.899379 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:23.899388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.899825 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.899829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:23.899902 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:23.900578 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:23.900593 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:23.902473 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
W20260924 16:12:23.902755 11294 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:23.897158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:47965 (local address 127.1.253.126:44947)
0924 16:12:23.897298 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:23.897302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:23.897433 (+ 131us) server_negotiation.cc:407] Connection header received
0924 16:12:23.897556 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:23.897558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:23.897604 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:23.897671 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:23.898109 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.898708 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.899477 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.899684 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.899770 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:23.902622 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:47965: BlockingRecv error: recv got EOF from 127.1.253.67:47965 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 16:12:24 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:24 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:24.526366 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:24.517183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:34161)
0924 16:12:24.517474 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:24.517487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:24.517581 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:24.517840 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:24.517842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:24.518018 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:24.518118 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:24.518123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:24.518962 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:24.518965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:24.519514 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:24.519522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:24.520026 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:24.520029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:24.520091 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:24.524406 (+ 4315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:24.524419 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:24.526228 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
W20260924 16:12:24.526643 11298 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:24.517255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:34161 (local address 127.1.253.126:44947)
0924 16:12:24.517559 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:24.517562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:24.517578 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:12:24.517666 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:24.517668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:24.517714 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:24.517800 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:24.518252 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:24.518856 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:24.519637 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:24.519911 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:24.520018 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:24.526456 (+ 6438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:34161: BlockingRecv error: recv got EOF from 127.1.253.65:34161 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":107,"threads_started":1}
Sep 24 16:12:24 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:24 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:24.718197 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:24.712391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:59815)
0924 16:12:24.712765 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:24.712787 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:24.712933 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:24.713242 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:24.713246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:24.713458 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:24.713560 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:24.713566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:24.714403 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:24.714409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:24.714981 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:24.714989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:24.715487 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:24.715491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:24.715577 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:24.716146 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:24.716159 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:24.717999 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":124,"threads_started":1}
W20260924 16:12:24.718293 11298 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:24.712513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:59815 (local address 127.1.253.126:44947)
0924 16:12:24.712729 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:24.712734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:24.712868 (+ 134us) server_negotiation.cc:407] Connection header received
0924 16:12:24.713052 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:24.713056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:24.713115 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:24.713211 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:24.713688 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:24.714284 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:24.715123 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:24.715366 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:24.715481 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:24.718144 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:59815: BlockingRecv error: recv got EOF from 127.1.253.66:59815 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 24 16:12:24 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:24 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:24.909631 11300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:24.903472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:41397)
0924 16:12:24.903892 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:24.903913 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:24.904050 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:24.904374 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:24.904377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:24.904652 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:24.904752 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:24.904758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:24.905737 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:24.905742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:24.906452 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:24.906461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:24.906918 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:24.906922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:24.907008 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:24.907579 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:24.907594 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:24.909452 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":94,"threads_started":1}
W20260924 16:12:24.909670 11298 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:24.903674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:41397 (local address 127.1.253.126:44947)
0924 16:12:24.903853 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:24.903858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:24.903990 (+ 132us) server_negotiation.cc:407] Connection header received
0924 16:12:24.904174 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:24.904178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:24.904263 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:24.904351 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:24.904871 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:24.905635 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:24.906570 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:24.906806 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:24.906915 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:24.909569 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:41397: BlockingRecv error: recv got EOF from 127.1.253.67:41397 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 16:12:25 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:25 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:25.532900 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:25.527309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:40855)
0924 16:12:25.527634 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:25.527648 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:25.527740 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:25.528006 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:25.528008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:25.528200 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:25.528296 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:25.528301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.529047 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.529050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:25.529628 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:25.529636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.530094 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.530099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:25.530195 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:25.530893 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:25.530906 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:25.532784 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":109,"threads_started":1}
W20260924 16:12:25.533067 11302 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:25.527414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:40855 (local address 127.1.253.126:44947)
0924 16:12:25.527693 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:25.527697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:25.527711 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:12:25.527830 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:25.527832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:25.527877 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:25.527952 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:25.528403 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.528923 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.529734 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.529958 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.530035 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:25.532925 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:40855: BlockingRecv error: recv got EOF from 127.1.253.65:40855 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
Sep 24 16:12:25 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:25 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:25.725322 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:25.719250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:42409)
0924 16:12:25.719554 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:25.719568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:25.719677 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:25.719930 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:25.719933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:25.720122 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:25.720218 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:25.720223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.721052 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.721055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:25.721660 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:25.721670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.722161 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.722164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:25.722311 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:25.722903 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:25.722916 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:25.725190 (+ 2274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":117,"threads_started":1}
W20260924 16:12:25.725646 11302 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:25.719461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:42409 (local address 127.1.253.126:44947)
0924 16:12:25.719644 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:25.719648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:25.719662 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:12:25.719770 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:25.719773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:25.719819 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:25.719893 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:25.720379 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.720914 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.721786 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.722011 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.722100 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:25.725361 (+ 3261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:42409: BlockingRecv error: recv got EOF from 127.1.253.66:42409 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 16:12:25 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:25 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:25.915668 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:25.910483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:36741)
0924 16:12:25.910787 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:25.910800 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:25.910894 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:25.911127 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:25.911129 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:25.911315 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:25.911409 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:25.911414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.912115 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.912118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:25.912685 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:25.912693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.913096 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.913099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:25.913165 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:25.913670 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:25.913683 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:25.915541 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":126,"threads_started":1}
W20260924 16:12:25.915788 11302 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:25.910580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:36741 (local address 127.1.253.126:44947)
0924 16:12:25.910723 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:25.910727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:25.910871 (+ 144us) server_negotiation.cc:407] Connection header received
0924 16:12:25.910969 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:25.910971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:25.911018 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:25.911087 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:25.911510 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.912018 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.912787 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.912995 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.913068 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:25.915659 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:36741: BlockingRecv error: recv got EOF from 127.1.253.67:36741 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 16:12:26 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:26 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:26.539379 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:26.533793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:53621)
0924 16:12:26.534109 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:26.534127 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:26.534299 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:26.534585 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:26.534589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:26.534805 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:26.534899 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:26.534905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:26.535675 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:26.535679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:26.536233 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:26.536241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:26.536699 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:26.536702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:26.536783 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:26.537309 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:26.537320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:26.539238 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":106,"threads_started":1}
W20260924 16:12:26.539508 11306 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:26.533834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:53621 (local address 127.1.253.126:44947)
0924 16:12:26.534110 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:26.534116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:26.534186 (+ 70us) server_negotiation.cc:407] Connection header received
0924 16:12:26.534409 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:26.534413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:26.534467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:26.534560 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:26.535023 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:26.535565 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:26.536354 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:26.536586 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:26.536695 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:26.539359 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:53621: BlockingRecv error: recv got EOF from 127.1.253.65:53621 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
Sep 24 16:12:26 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:26 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:26.733136 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:26.726298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:42227)
0924 16:12:26.726595 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:26.726611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:26.726725 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:26.726982 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:26.726986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:26.727253 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:26.727399 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:26.727408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:26.728157 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:26.728163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:26.729203 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:26.729219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:26.729669 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:26.729674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:26.729785 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:26.730514 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:26.730533 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:26.732928 (+ 2395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":114,"threads_started":1}
W20260924 16:12:26.733177 11306 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:26.726354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:42227 (local address 127.1.253.126:44947)
0924 16:12:26.726503 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:26.726507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:26.726688 (+ 181us) server_negotiation.cc:407] Connection header received
0924 16:12:26.726800 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:26.726802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:26.726848 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:26.726921 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:26.727524 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:26.728046 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:26.729337 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:26.729554 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:26.729627 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:26.733080 (+ 3453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:42227: BlockingRecv error: recv got EOF from 127.1.253.66:42227 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 16:12:26 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:26 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:26.922753 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:26.916727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:46715)
0924 16:12:26.917106 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:26.917129 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:26.917283 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:26.917578 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:26.917581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:26.917799 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:26.917897 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:26.917902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:26.918756 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:26.918762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:26.919324 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:26.919333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:26.919793 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:26.919797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:26.919885 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:26.920425 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:26.920437 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:26.922586 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":128,"threads_started":1}
W20260924 16:12:26.922881 11306 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:26.916896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:46715 (local address 127.1.253.126:44947)
0924 16:12:26.917062 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:26.917067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:26.917206 (+ 139us) server_negotiation.cc:407] Connection header received
0924 16:12:26.917393 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:26.917397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:26.917457 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:26.917551 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:26.918021 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:26.918638 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:26.919450 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:26.919676 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:26.919787 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:26.922742 (+ 2955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:46715: BlockingRecv error: recv got EOF from 127.1.253.67:46715 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 16:12:27 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:27 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:27.545544 11309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:27.540057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:45279)
0924 16:12:27.540400 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:27.540413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:27.540504 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:27.540736 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:27.540738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:27.540917 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:27.541016 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:27.541022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.541810 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.541813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:27.542444 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:27.542454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.542938 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.542941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:27.543001 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:27.543506 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:27.543518 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:27.545369 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":93,"threads_started":1}
W20260924 16:12:27.545650 11310 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:27.540263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:45279 (local address 127.1.253.126:44947)
0924 16:12:27.540520 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:27.540523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:27.540539 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:12:27.540574 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:27.540576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:27.540620 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:27.540698 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:27.541153 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.541713 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.542600 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.542831 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.542922 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:27.545489 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:45279: BlockingRecv error: recv got EOF from 127.1.253.65:45279 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
Sep 24 16:12:27 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:27 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:27.739961 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:27.734084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:52729)
0924 16:12:27.734447 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:27.734461 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:27.734568 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:27.734809 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:27.734811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:27.734991 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:27.735085 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:27.735089 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.735837 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.735839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:27.736467 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:27.736478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.737045 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.737048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:27.737164 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:27.737786 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:27.737801 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:27.739841 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
W20260924 16:12:27.740093 11310 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:27.734132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:52729 (local address 127.1.253.126:44947)
0924 16:12:27.734340 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:27.734346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:27.734565 (+ 219us) server_negotiation.cc:407] Connection header received
0924 16:12:27.734655 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:27.734657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:27.734708 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:27.734772 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:27.735216 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.735745 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.736588 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.736908 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.737031 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:27.739975 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:52729: BlockingRecv error: recv got EOF from 127.1.253.66:52729 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 16:12:27 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:27 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:27.929706 11312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:27.923788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:55173)
0924 16:12:27.924158 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:27.924172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:27.924274 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:27.924606 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:27.924608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:27.924785 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:27.924883 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:27.924888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.925657 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.925659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:27.926214 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:27.926223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.926845 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.926849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:27.926956 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:27.927569 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:27.927582 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:27.929589 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
W20260924 16:12:27.929894 11310 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:27.923860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:55173 (local address 127.1.253.126:44947)
0924 16:12:27.924008 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:27.924012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:27.924244 (+ 232us) server_negotiation.cc:407] Connection header received
0924 16:12:27.924408 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:27.924411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:27.924469 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:27.924570 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:27.924995 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.925542 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.926401 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.926697 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.926809 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:27.929734 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:55173: BlockingRecv error: recv got EOF from 127.1.253.67:55173 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 16:12:28 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:28 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:28.552183 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:28.546396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:40755)
0924 16:12:28.546725 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:28.546741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:28.546845 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:28.547141 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:28.547144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:28.547331 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:28.547432 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:28.547438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.548183 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.548186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:28.548783 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:28.548792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.549225 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.549228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:28.549293 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:28.549969 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:28.549982 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:28.552001 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":114,"threads_started":1}
W20260924 16:12:28.552330 11314 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:28.546562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:40755 (local address 127.1.253.126:44947)
0924 16:12:28.546780 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:28.546784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:28.546798 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:12:28.546945 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:28.546947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:28.546995 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:28.547070 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:28.547565 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.548081 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.548892 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.549110 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.549185 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:28.552198 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:40755: BlockingRecv error: recv got EOF from 127.1.253.65:40755 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":63,"threads_started":1}
Sep 24 16:12:28 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:28 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:28.746803 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:28.740989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:36813)
0924 16:12:28.741291 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:28.741309 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:28.741437 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:28.741749 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:28.741753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:28.741963 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:28.742064 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:28.742070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.743063 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.743077 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:28.743650 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:28.743658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.744129 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.744132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:28.744223 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:28.744782 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:28.744803 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:28.746621 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":87,"threads_started":1}
W20260924 16:12:28.746863 11314 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:28.741053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:36813 (local address 127.1.253.126:44947)
0924 16:12:28.741231 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:28.741237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:28.741384 (+ 147us) server_negotiation.cc:407] Connection header received
0924 16:12:28.741567 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:28.741573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:28.741625 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:28.741714 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:28.742185 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.742922 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.743777 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.744011 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.744123 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:28.746759 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:36813: BlockingRecv error: recv got EOF from 127.1.253.66:36813 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 16:12:28 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:28 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:28.936139 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:28.930615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:54105)
0924 16:12:28.930901 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:28.930915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:28.931011 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:28.931278 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:28.931281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:28.931506 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:28.931609 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:28.931614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.932362 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.932364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:28.932907 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:28.932915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.933430 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.933436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:28.933508 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:28.934108 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:28.934120 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:28.936024 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":116,"threads_started":1}
W20260924 16:12:28.936293 11314 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:28.930718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:54105 (local address 127.1.253.126:44947)
0924 16:12:28.930865 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:28.930869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:28.930982 (+ 113us) server_negotiation.cc:407] Connection header received
0924 16:12:28.931099 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:28.931102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:28.931158 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:28.931228 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:28.931714 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.932247 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.933078 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.933306 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.933397 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:28.936164 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:54105: BlockingRecv error: recv got EOF from 127.1.253.67:54105 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 16:12:29 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:29 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:29.571985 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:29.553165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:38371)
0924 16:12:29.553511 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:29.553525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:29.553638 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:29.554040 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:29.554044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:29.554343 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:29.554444 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:29.554450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:29.555181 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:29.555183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:29.555939 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:29.555953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:29.556423 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:29.556427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:29.556515 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:29.569591 (+ 13076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:29.569615 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:29.571829 (+ 2214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":92,"threads_started":1}
W20260924 16:12:29.572190 11318 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:29.553236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:38371 (local address 127.1.253.126:44947)
0924 16:12:29.553744 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:29.553749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:29.553763 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:12:29.553811 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:29.553814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:29.553879 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:29.553975 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:29.554553 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:29.555081 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:29.556058 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:29.556301 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:29.556418 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:29.572026 (+ 15608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:38371: BlockingRecv error: recv got EOF from 127.1.253.65:38371 (error 108)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":63,"threads_started":1}
Sep 24 16:12:29 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:29 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:29.752996 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:29.747700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:36523)
0924 16:12:29.747993 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:29.748008 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:29.748121 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:29.748369 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:29.748370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:29.748551 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:29.748652 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:29.748656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:29.749392 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:29.749395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:29.749955 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:29.749962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:29.750406 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:29.750410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:29.750471 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:29.750989 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:29.751000 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:29.752846 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":96,"threads_started":1}
W20260924 16:12:29.753144 11318 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:29.747773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:36523 (local address 127.1.253.126:44947)
0924 16:12:29.747937 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:29.747941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:29.748123 (+ 182us) server_negotiation.cc:407] Connection header received
0924 16:12:29.748191 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:29.748193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:29.748243 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:29.748328 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:29.748760 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:29.749299 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:29.750068 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:29.750308 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:29.750388 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:29.753014 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:36523: BlockingRecv error: recv got EOF from 127.1.253.66:36523 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 16:12:29 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:29 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:29.942422 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:29.936901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:33353)
0924 16:12:29.937169 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:29.937183 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:29.937296 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:29.937527 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:29.937529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:29.937708 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:29.937805 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:29.937810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:29.938692 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:29.938695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:29.939318 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:29.939328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:29.939760 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:29.939764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:29.939822 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:29.940324 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:29.940335 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:29.942227 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
W20260924 16:12:29.942524 11318 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:29.936997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:33353 (local address 127.1.253.126:44947)
0924 16:12:29.937174 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:29.937178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:29.937261 (+ 83us) server_negotiation.cc:407] Connection header received
0924 16:12:29.937372 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:29.937375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:29.937423 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:29.937489 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:29.937899 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:29.938593 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:29.939430 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:29.939662 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:29.939742 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:29.942397 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:33353: BlockingRecv error: recv got EOF from 127.1.253.67:33353 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 16:12:30 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:30 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:30.578179 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:30.572793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:44173)
0924 16:12:30.573098 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:30.573125 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:30.573224 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:30.573476 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:30.573478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:30.573690 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:30.573792 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:30.573797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.574622 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.574625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:30.575194 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:30.575202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.575609 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.575612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:30.575675 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:30.576165 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:30.576177 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:30.578005 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":71,"threads_started":1}
W20260924 16:12:30.578302 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:30.572940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:44173 (local address 127.1.253.126:44947)
0924 16:12:30.573206 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:30.573210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:30.573231 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:12:30.573307 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:30.573309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:30.573352 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:30.573428 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:30.573894 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.574505 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.575297 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.575510 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.575586 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:30.578135 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:44173: BlockingRecv error: recv got EOF from 127.1.253.65:44173 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":87,"threads_started":1}
Sep 24 16:12:30 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:30 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:30.759593 11323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:30.754003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:44739)
0924 16:12:30.754339 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:30.754353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:30.754461 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:30.754708 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:30.754711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:30.754896 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:30.754994 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:30.754999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.755785 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.755788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:30.756340 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:30.756348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.756801 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.756804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:30.756866 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:30.757459 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:30.757471 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:30.759464 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":119,"threads_started":1}
W20260924 16:12:30.759714 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:30.754197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:44739 (local address 127.1.253.126:44947)
0924 16:12:30.754341 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:30.754346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:30.754433 (+ 87us) server_negotiation.cc:407] Connection header received
0924 16:12:30.754544 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:30.754547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:30.754593 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:30.754668 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:30.755103 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.755660 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.756462 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.756699 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.756780 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:30.759598 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:44739: BlockingRecv error: recv got EOF from 127.1.253.66:44739 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 16:12:30 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:30 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:30.949111 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:30.943432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:33999)
0924 16:12:30.943725 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:30.943740 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:30.943832 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:30.944088 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:30.944090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:30.944335 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:30.944473 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:30.944480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.945422 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.945425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:30.946162 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:30.946172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.946625 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.946628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:30.946689 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:30.947188 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:30.947199 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:30.948992 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
W20260924 16:12:30.949249 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:30.943581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:33999 (local address 127.1.253.126:44947)
0924 16:12:30.943748 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:30.943752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:30.943794 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:12:30.943920 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:30.943923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:30.943967 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:30.944039 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:30.944630 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.945261 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.946304 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.946521 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.946603 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:30.949128 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:33999: BlockingRecv error: recv got EOF from 127.1.253.67:33999 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 16:12:31 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:31 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:31.584656 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:31.579128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:47777)
0924 16:12:31.579460 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:31.579484 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:31.579650 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:31.579980 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:31.579984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:31.580208 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:31.580331 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:31.580337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:31.581072 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:31.581075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:31.581633 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:31.581640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:31.582081 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:31.582085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:31.582157 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:31.582700 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:31.582712 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:31.584491 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
W20260924 16:12:31.584739 11326 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:31.579231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:47777 (local address 127.1.253.126:44947)
0924 16:12:31.579502 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:31.579507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:31.579597 (+ 90us) server_negotiation.cc:407] Connection header received
0924 16:12:31.579776 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:31.579779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:31.579835 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:31.579929 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:31.580441 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:31.580974 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:31.581738 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:31.581942 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:31.582015 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:31.584606 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:47777: BlockingRecv error: recv got EOF from 127.1.253.65:47777 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":59,"threads_started":1}
Sep 24 16:12:31 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:31 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:31.772449 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:31.760644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:42113)
0924 16:12:31.760939 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:31.760952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:31.761050 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:31.761275 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:31.761277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:31.761472 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:31.761620 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:31.761625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:31.762357 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:31.762359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:31.762945 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:31.762953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:31.763355 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:31.763358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:31.763420 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:31.770215 (+ 6795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:31.770298 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:31.772244 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":108,"threads_started":1}
W20260924 16:12:31.772476 11326 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:31.760751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:42113 (local address 127.1.253.126:44947)
0924 16:12:31.760923 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:31.760927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:31.761006 (+ 79us) server_negotiation.cc:407] Connection header received
0924 16:12:31.761118 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:31.761120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:31.761164 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:31.761245 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:31.761728 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:31.762269 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:31.763049 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:31.763262 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:31.763335 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:31.772371 (+ 9036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:42113: BlockingRecv error: recv got EOF from 127.1.253.66:42113 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 16:12:31 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:31 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:31.955475 11328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:31.950045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:38473)
0924 16:12:31.950345 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:31.950368 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:31.950471 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:31.950719 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:31.950721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:31.950908 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:31.951005 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:31.951011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:31.951726 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:31.951729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:31.952276 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:31.952284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:31.952680 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:31.952682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:31.952742 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:31.953242 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:31.953254 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:31.955283 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":110,"threads_started":1}
W20260924 16:12:31.955597 11326 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:31.950084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:38473 (local address 127.1.253.126:44947)
0924 16:12:31.950273 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:31.950280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:31.950440 (+ 160us) server_negotiation.cc:407] Connection header received
0924 16:12:31.950553 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:31.950555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:31.950610 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:31.950679 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:31.951106 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:31.951628 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:31.952380 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:31.952587 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:31.952661 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:31.955468 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:38473: BlockingRecv error: recv got EOF from 127.1.253.67:38473 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 16:12:32 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:32 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:32.591305 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:32.585529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:40719)
0924 16:12:32.585857 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:32.585870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:32.585965 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:32.586294 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:32.586296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:32.586476 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:32.586578 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:32.586583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.587332 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.587335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:32.587887 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:32.587895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.588497 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.588501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:32.588567 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:32.589251 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:32.589269 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:32.591181 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":89,"threads_started":1}
W20260924 16:12:32.591548 11330 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:32.585691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:40719 (local address 127.1.253.126:44947)
0924 16:12:32.585929 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:32.585932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:32.585946 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:12:32.586068 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:32.586071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:32.586121 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:32.586199 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:32.586683 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.587235 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.587996 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.588367 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.588472 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:32.591329 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:40719: BlockingRecv error: recv got EOF from 127.1.253.65:40719 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
Sep 24 16:12:32 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:32 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:32.779086 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:32.773292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:54883)
0924 16:12:32.773561 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:32.773574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:32.773668 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:32.773940 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:32.773942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:32.774126 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:32.774226 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:32.774305 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.775109 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.775112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:32.775665 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:32.775673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.776085 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.776088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:32.776147 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:32.776741 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:32.776754 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:32.778966 (+ 2212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
W20260924 16:12:32.779281 11330 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:32.773416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:54883 (local address 127.1.253.126:44947)
0924 16:12:32.773601 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:32.773605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:32.773620 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:12:32.773756 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:32.773759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:32.773807 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:32.773902 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:32.774459 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.775010 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.775769 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.775986 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.776068 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:32.779132 (+ 3064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:54883: BlockingRecv error: recv got EOF from 127.1.253.66:54883 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 24 16:12:32 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:32 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:32.962538 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:32.956540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:46867)
0924 16:12:32.957086 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:32.957102 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:32.957226 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:32.957528 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:32.957530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:32.957706 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:32.957808 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:32.957813 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.958660 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.958663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:32.959223 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:32.959239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.959721 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.959724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:32.959785 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:32.960370 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:32.960391 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:32.962357 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":84,"threads_started":1}
W20260924 16:12:32.962606 11330 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:32.956716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:46867 (local address 127.1.253.126:44947)
0924 16:12:32.956890 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:32.956897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:32.957204 (+ 307us) server_negotiation.cc:407] Connection header received
0924 16:12:32.957357 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:32.957360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:32.957412 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:32.957487 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:32.957912 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.958557 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.959358 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.959615 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.959699 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:32.962495 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:46867: BlockingRecv error: recv got EOF from 127.1.253.67:46867 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 16:12:33 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:33 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:33.597934 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:33.592221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:52229)
0924 16:12:33.592567 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:33.592580 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:33.592687 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:33.592950 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:33.592952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:33.593131 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:33.593229 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:33.593234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.593960 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.593962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:33.594617 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:33.594627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.595033 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.595036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:33.595098 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:33.595607 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:33.595617 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:33.597806 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":99,"threads_started":1}
W20260924 16:12:33.598117 11334 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:33.592299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:52229 (local address 127.1.253.126:44947)
0924 16:12:33.592578 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:33.592582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:33.592639 (+ 57us) server_negotiation.cc:407] Connection header received
0924 16:12:33.592780 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:33.592782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:33.592826 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:33.592904 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:33.593334 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.593865 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.594720 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.594935 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.595011 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:33.597936 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:52229: BlockingRecv error: recv got EOF from 127.1.253.65:52229 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":67,"threads_started":1}
Sep 24 16:12:33 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:33 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:33.786070 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:33.779995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:49029)
0924 16:12:33.780359 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:33.780378 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:33.780545 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:33.780868 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:33.780871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:33.781105 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:33.781209 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:33.781215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.782014 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.782020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:33.782774 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:33.782786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.783285 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.783290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:33.783385 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:33.784019 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:33.784033 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:33.785933 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":111,"threads_started":1}
W20260924 16:12:33.786196 11334 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:33.780224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:49029 (local address 127.1.253.126:44947)
0924 16:12:33.780396 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:33.780404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:33.780443 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:12:33.780671 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:33.780677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:33.780739 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:33.780843 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:33.781343 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.781894 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.782914 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.783158 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.783284 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:33.786075 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:49029: BlockingRecv error: recv got EOF from 127.1.253.66:49029 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 16:12:33 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:33 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:33.969436 11336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:33.963580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:48281)
0924 16:12:33.963910 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:33.963923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:33.964070 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:33.964378 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:33.964381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:33.964602 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:33.964704 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:33.964709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.965473 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.965478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:33.966091 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:33.966100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.966618 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.966622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:33.966710 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:33.967226 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:33.967239 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:33.969248 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":78,"threads_started":1}
W20260924 16:12:33.969520 11334 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:33.963662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:48281 (local address 127.1.253.126:44947)
0924 16:12:33.963810 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:33.963815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:33.964001 (+ 186us) server_negotiation.cc:407] Connection header received
0924 16:12:33.964185 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:33.964190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:33.964274 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:33.964354 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:33.964827 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.965356 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.966218 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.966487 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.966612 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:33.969396 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:48281: BlockingRecv error: recv got EOF from 127.1.253.67:48281 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 16:12:34 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:34 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:34.618759 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:34.598883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:47387)
0924 16:12:34.599180 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:34.599194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:34.599305 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:34.599774 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:34.599778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:34.600001 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:34.600104 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:34.600111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.601050 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.601056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:34.601640 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:34.601649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.602183 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.602186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:34.602317 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:34.616611 (+ 14294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:34.616627 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:34.618618 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":97,"threads_started":1}
W20260924 16:12:34.618969 11338 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:34.598973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:47387 (local address 127.1.253.126:44947)
0924 16:12:34.599315 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:34.599319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:34.599355 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:12:34.599395 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:34.599398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:34.599441 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:34.599540 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:34.600231 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.600935 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.601777 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.602049 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.602162 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:34.618812 (+ 16650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:47387: BlockingRecv error: recv got EOF from 127.1.253.65:47387 (error 108)
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":78,"threads_started":1}
Sep 24 16:12:34 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:34 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:34.792665 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:34.787161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:56249)
0924 16:12:34.787449 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:34.787462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:34.787566 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:34.787801 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:34.787803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:34.787979 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:34.788081 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:34.788086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.788801 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.788803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:34.789588 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:34.789600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.790130 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.790134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:34.790219 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:34.790721 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:34.790732 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:34.792490 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":98,"threads_started":1}
W20260924 16:12:34.792759 11338 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:34.787224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:56249 (local address 127.1.253.126:44947)
0924 16:12:34.787375 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:34.787379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:34.787533 (+ 154us) server_negotiation.cc:407] Connection header received
0924 16:12:34.787647 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:34.787649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:34.787695 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:34.787765 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:34.788174 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.788685 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.789712 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.790030 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.790127 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:34.792636 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:56249: BlockingRecv error: recv got EOF from 127.1.253.66:56249 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 16:12:34 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:34 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.67@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:34.975842 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:34.970434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.67:36193)
0924 16:12:34.970717 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:34.970730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:34.970829 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:34.971069 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:34.971071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:34.971266 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:34.971364 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:34.971369 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.972117 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.972120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:34.972686 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:34.972694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.973095 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.973098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:34.973157 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:34.973747 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:34.973759 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:34.975715 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
W20260924 16:12:34.976027 11338 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:34.970506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:36193 (local address 127.1.253.126:44947)
0924 16:12:34.970639 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:34.970643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:34.970797 (+ 154us) server_negotiation.cc:407] Connection header received
0924 16:12:34.970909 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:34.970911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:34.970960 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:34.971031 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:34.971471 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.972010 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.972787 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.973000 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.973088 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:34.975848 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.67:36193: BlockingRecv error: recv got EOF from 127.1.253.67:36193 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 16:12:35 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:35 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.65@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:35.625281 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:35.619712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.65:48737)
0924 16:12:35.619988 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:35.620006 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:35.620113 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:35.620439 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:35.620442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:35.620655 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:35.620794 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:35.620802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:35.621532 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:35.621535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:35.622341 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:35.622352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:35.622785 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:35.622788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:35.622851 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:35.623348 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:35.623359 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:35.625158 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
W20260924 16:12:35.625550 11342 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:35.619794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:48737 (local address 127.1.253.126:44947)
0924 16:12:35.620039 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:35.620043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:35.620059 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:12:35.620227 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:35.620230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:35.620293 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:35.620373 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:35.620909 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:35.621437 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:35.622472 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:35.622689 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:35.622775 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:35.625287 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.65:48737: BlockingRecv error: recv got EOF from 127.1.253.65:48737 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
Sep 24 16:12:35 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:12:35 dist-test-slave-6wvm krb5kdc[10284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.253.66@KRBTEST.COM for oryx/127.1.253.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:12:35.799511 11343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:35.793827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44947 (local address 127.1.253.66:36579)
0924 16:12:35.794103 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:35.794117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:35.794209 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:35.794480 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:35.794483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:35.794673 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:35.794772 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:35.794777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:35.795511 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:35.795514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:35.796150 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:35.796162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:35.796702 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:35.796707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:35.796814 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:35.797413 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:35.797425 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:35.799338 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.253.126:44947: Server oryx/127.1.253.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
W20260924 16:12:35.799553 11342 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:35.793956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:36579 (local address 127.1.253.126:44947)
0924 16:12:35.794144 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:35.794147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:35.794165 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:12:35.794323 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:35.794325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:35.794369 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:35.794438 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:35.794876 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:35.795395 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:35.796285 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:35.796553 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:35.796661 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:35.799454 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.253.66:36579: BlockingRecv error: recv got EOF from 127.1.253.66:36579 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 16:12:35.865967 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 10867
I20260924 16:12:35.872989 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 11019
I20260924 16:12:35.879215 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 11151
I20260924 16:12:35.885118 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 10797
2026-09-24T16:12:35Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18108 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T16:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:35Z Disabled control of system clock
I20260924 16:12:35.916637 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:44707
--webserver_interface=127.1.253.126
--webserver_port=0
--builtin_ntp_servers=127.1.253.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:44707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 16:12:36.030226 11347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:36.030539 11347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:36.030572 11347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:36.034646 11347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:36.034695 11347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:36.034710 11347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:36.034726 11347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:36.034742 11347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:36.039290 11347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:44707
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:44707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:36.040323 11347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:36.041750 11347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:12:36.050632 11347 server_base.cc:1034] running on GCE node
W20260924 16:12:36.050679 11352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:36.050781 11355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:36.050683 11353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:36.051430 11347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:36.052520 11347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:36.053702 11347 hybrid_clock.cc:648] HybridClock initialized: now 1790266356053672 us; error 42 us; skew 500 ppm
I20260924 16:12:36.055635 11347 webserver.cc:460] Webserver started at http://127.1.253.126:45501/ using document root <none> and password file <none>
I20260924 16:12:36.056218 11347 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:36.056317 11347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:36.056553 11347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:36.058627 11347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "5159771f6ba648779f9673efa494232e"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "c1437053c6ef2fd2b687deb832e2ec60"
server_key_iv: "2f1f59966afb76f786fbeb648b980ca2"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.059183 11347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "5159771f6ba648779f9673efa494232e"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "c1437053c6ef2fd2b687deb832e2ec60"
server_key_iv: "2f1f59966afb76f786fbeb648b980ca2"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.062868 11347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:12:36.065399 11361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.066656 11347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 16:12:36.066797 11347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "5159771f6ba648779f9673efa494232e"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "c1437053c6ef2fd2b687deb832e2ec60"
server_key_iv: "2f1f59966afb76f786fbeb648b980ca2"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.066911 11347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:36.084419 11347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:36.085188 11347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:36.085462 11347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:36.094074 11347 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:44707
I20260924 16:12:36.094075 11413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:44707 every 8 connection(s)
I20260924 16:12:36.095153 11347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:36.098050 11414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:36.102447 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 11347
I20260924 16:12:36.102538 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:36.102831 2037 external_mini_cluster.cc:1428] Setting key eb695a79ecc505f89cadf49218c8c64a
I20260924 16:12:36.103899 11414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e: Bootstrap starting.
I20260924 16:12:36.106300 11414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:36.107158 11414 log.cc:824] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:36.109155 11414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e: No bootstrap required, opened a new log
I20260924 16:12:36.112319 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:36.104219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.1.253.126:44707)
0924 16:12:36.104819 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:36.104835 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:36.104869 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:12:36.105499 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:36.105515 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:36.105895 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:36.106164 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:36.106645 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.107653 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.108612 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.109290 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.109450 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:36.110264 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:36.110271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:36.110288 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:36.110773 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:36.110843 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:36.110869 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:36.110969 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:36.111097 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 16:12:36.111382 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":124,"threads_started":1}
I20260924 16:12:36.113605 11414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5159771f6ba648779f9673efa494232e" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44707 } }
I20260924 16:12:36.113829 11414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:36.113890 11414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5159771f6ba648779f9673efa494232e, State: Initialized, Role: FOLLOWER
I20260924 16:12:36.114351 11414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5159771f6ba648779f9673efa494232e" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44707 } }
I20260924 16:12:36.114470 11414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:36.114576 11414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:36.114710 11414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:36.115541 11414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5159771f6ba648779f9673efa494232e" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44707 } }
I20260924 16:12:36.115828 11414 leader_election.cc:302] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5159771f6ba648779f9673efa494232e; no voters:
I20260924 16:12:36.116158 11414 leader_election.cc:288] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:36.116312 11419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:36.116582 11419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [term 1 LEADER]: Becoming Leader. State: Replica: 5159771f6ba648779f9673efa494232e, State: Running, Role: LEADER
I20260924 16:12:36.116933 11419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5159771f6ba648779f9673efa494232e" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44707 } }
I20260924 16:12:36.117447 11414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:36.118711 11421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5159771f6ba648779f9673efa494232e. Latest consensus state: current_term: 1 leader_uuid: "5159771f6ba648779f9673efa494232e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5159771f6ba648779f9673efa494232e" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44707 } } }
I20260924 16:12:36.118742 11420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5159771f6ba648779f9673efa494232e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5159771f6ba648779f9673efa494232e" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 44707 } } }
I20260924 16:12:36.118834 11421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:36.118844 11420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:36.119246 11428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:36.121816 11428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:36.126874 11428 catalog_manager.cc:1347] Generated new cluster ID: 98e2f149389b4b4bb0a89a1979e6ca2c
I20260924 16:12:36.126940 11428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:36.157469 11428 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:36.158835 11428 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:36.171885 11428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5159771f6ba648779f9673efa494232e: Generated new TSK 0
I20260924 16:12:36.172719 11428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:12:36.179414 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:44707
--builtin_ntp_servers=127.1.253.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 16:12:36.291079 11438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:36.291312 11438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:36.291345 11438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:36.295327 11438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:36.295406 11438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:36.295496 11438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:12:36.300120 11438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:36.301205 11438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:36.302659 11438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:36.312382 11446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:36.312389 11443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:36.312529 11444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:36.312546 11438 server_base.cc:1034] running on GCE node
I20260924 16:12:36.313014 11438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:36.313663 11438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:36.314854 11438 hybrid_clock.cc:648] HybridClock initialized: now 1790266356314839 us; error 24 us; skew 500 ppm
I20260924 16:12:36.316733 11438 webserver.cc:460] Webserver started at http://127.1.253.65:45725/ using document root <none> and password file <none>
I20260924 16:12:36.317381 11438 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:36.317474 11438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:36.317684 11438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:36.319979 11438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "1c9d97c34428493788479d934faf90aa"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "016c6a7ceddf20e6fcf7c6df49b184d1"
server_key_iv: "077f698f359a611d6e7573588d83f0d8"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.320564 11438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "1c9d97c34428493788479d934faf90aa"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "016c6a7ceddf20e6fcf7c6df49b184d1"
server_key_iv: "077f698f359a611d6e7573588d83f0d8"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.324146 11438 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 16:12:36.326610 11452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.327767 11438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:36.327905 11438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "1c9d97c34428493788479d934faf90aa"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "016c6a7ceddf20e6fcf7c6df49b184d1"
server_key_iv: "077f698f359a611d6e7573588d83f0d8"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.328015 11438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:36.344849 11438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:36.345572 11438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:36.345777 11438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:36.346588 11438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:36.347571 11438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:36.347645 11438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.347712 11438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:36.347764 11438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.356531 11438 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:40219
I20260924 16:12:36.356554 11565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:40219 every 8 connection(s)
I20260924 16:12:36.357448 11438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:36.366730 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 11438
I20260924 16:12:36.366791 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:36.359376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:50059 (local address 127.1.253.126:44707)
0924 16:12:36.359519 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:36.359523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:36.360581 (+ 1058us) server_negotiation.cc:407] Connection header received
0924 16:12:36.362052 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:36.362055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:36.362118 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:36.362219 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:36.363169 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.363703 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.364865 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.365090 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.365206 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:36.366278 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:36.366284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:36.366291 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:36.366304 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:36.366321 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:36.366329 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:36.366411 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:36.366580 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 16:12:36.366643 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:12:36.366886 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:36.367199 2037 external_mini_cluster.cc:1428] Setting key 2b464056c7f50accd6ddecf5639baefb
I20260924 16:12:36.367689 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:36.359779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44707 (local address 127.1.253.65:50059)
0924 16:12:36.360434 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:36.360474 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:36.361819 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:36.362434 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:36.362441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:36.362528 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:36.363018 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:36.363031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.363841 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.363851 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:36.364726 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:36.364735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.365250 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.365254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:36.365421 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:36.365995 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:36.365997 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:36.366037 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:36.366100 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:36.366412 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:36.366425 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:36.366433 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 16:12:36.366565 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 16:12:36.366853 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260924 16:12:36.368898 11566 heartbeater.cc:347] Connected to a master server at 127.1.253.126:44707
I20260924 16:12:36.369158 11566 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:36.369304 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:44707
--builtin_ntp_servers=127.1.253.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 16:12:36.369809 11566 heartbeater.cc:511] Master 127.1.253.126:44707 requested a full tablet report, sending...
I20260924 16:12:36.371559 11373 ts_manager.cc:194] Registered new tserver with Master: 1c9d97c34428493788479d934faf90aa (127.1.253.65:40219)
I20260924 16:12:36.373190 11373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.253.65:50059
W20260924 16:12:36.486059 11569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:36.486315 11569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:36.486351 11569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:36.490240 11569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:36.490314 11569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:36.490365 11569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:12:36.494912 11569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:36.495890 11569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:36.497247 11569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:36.506935 11574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:36.507086 11569 server_base.cc:1034] running on GCE node
W20260924 16:12:36.506964 11577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:36.506985 11575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:36.507695 11569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:36.508404 11569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:36.509590 11569 hybrid_clock.cc:648] HybridClock initialized: now 1790266356509559 us; error 49 us; skew 500 ppm
I20260924 16:12:36.511523 11569 webserver.cc:460] Webserver started at http://127.1.253.66:36407/ using document root <none> and password file <none>
I20260924 16:12:36.512135 11569 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:36.512228 11569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:36.512465 11569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:36.514609 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "a5aa02a398274fd4ba4f806f02959d46"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "545cc6c320035641848d2ad5e3deb2e2"
server_key_iv: "67a171ce127906ab4cf57d35ae7f5ef3"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.515178 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "a5aa02a398274fd4ba4f806f02959d46"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "545cc6c320035641848d2ad5e3deb2e2"
server_key_iv: "67a171ce127906ab4cf57d35ae7f5ef3"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.518999 11569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:12:36.521617 11583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.522907 11569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 16:12:36.523046 11569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "a5aa02a398274fd4ba4f806f02959d46"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "545cc6c320035641848d2ad5e3deb2e2"
server_key_iv: "67a171ce127906ab4cf57d35ae7f5ef3"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.523176 11569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:36.534853 11569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:36.535506 11569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:36.535701 11569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:36.536267 11569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:36.537216 11569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:36.537286 11569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.537359 11569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:36.537401 11569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.545922 11569 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:43433
I20260924 16:12:36.545943 11696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:43433 every 8 connection(s)
I20260924 16:12:36.546953 11569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:36.555934 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:36.548842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:48407 (local address 127.1.253.126:44707)
0924 16:12:36.548995 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:36.549000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:36.550112 (+ 1112us) server_negotiation.cc:407] Connection header received
0924 16:12:36.551566 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:36.551569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:36.551617 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:36.551692 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:36.552537 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.553051 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.554137 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.554418 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.554515 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:36.555526 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:36.555529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:36.555531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:36.555543 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:36.555561 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:36.555568 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:36.555624 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:36.555772 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 16:12:36.555821 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:12:36.556723 11699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:36.549293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44707 (local address 127.1.253.66:48407)
0924 16:12:36.549966 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:36.550006 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:36.551373 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:36.551861 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:36.551868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:36.551952 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:36.552422 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:36.552433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.553157 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.553160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:36.554035 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:36.554046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.554592 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.554595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:36.554733 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:36.555281 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:36.555282 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:36.555319 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:36.555386 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:36.555641 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:36.555648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:36.555652 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:12:36.555744 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 16:12:36.555949 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":111,"threads_started":1}
I20260924 16:12:36.556789 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 11569
I20260924 16:12:36.556902 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:36.557199 2037 external_mini_cluster.cc:1428] Setting key 7e76ece90a297c6baea700ffc9f498c8
I20260924 16:12:36.558094 11697 heartbeater.cc:347] Connected to a master server at 127.1.253.126:44707
I20260924 16:12:36.558370 11697 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:36.558974 11697 heartbeater.cc:511] Master 127.1.253.126:44707 requested a full tablet report, sending...
I20260924 16:12:36.559533 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:44707
--builtin_ntp_servers=127.1.253.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 16:12:36.560094 11373 ts_manager.cc:194] Registered new tserver with Master: a5aa02a398274fd4ba4f806f02959d46 (127.1.253.66:43433)
I20260924 16:12:36.560669 11373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.253.66:48407
W20260924 16:12:36.672660 11700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:36.672930 11700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:36.672966 11700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:36.676947 11700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:36.676994 11700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:36.677045 11700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:12:36.681630 11700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:36.682693 11700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:36.684001 11700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:36.693799 11708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:36.693785 11705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:36.693856 11700 server_base.cc:1034] running on GCE node
W20260924 16:12:36.693790 11706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:36.694531 11700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:36.695092 11700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:36.696249 11700 hybrid_clock.cc:648] HybridClock initialized: now 1790266356696216 us; error 43 us; skew 500 ppm
I20260924 16:12:36.698056 11700 webserver.cc:460] Webserver started at http://127.1.253.67:39403/ using document root <none> and password file <none>
I20260924 16:12:36.698741 11700 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:36.698850 11700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:36.699154 11700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:36.701268 11700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "c985c57ed7f84af99358d631a521bac7"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "d025b96988186b147afabea61978f699"
server_key_iv: "f3dd511364c0994a103c21fbf6d86fcb"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.701862 11700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "c985c57ed7f84af99358d631a521bac7"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "d025b96988186b147afabea61978f699"
server_key_iv: "f3dd511364c0994a103c21fbf6d86fcb"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.705932 11700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 16:12:36.708647 11714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.709949 11700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:36.710079 11700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "c985c57ed7f84af99358d631a521bac7"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "d025b96988186b147afabea61978f699"
server_key_iv: "f3dd511364c0994a103c21fbf6d86fcb"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.710191 11700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:36.729244 11700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:36.729935 11700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:36.730129 11700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:36.730769 11700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:36.731761 11700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:36.731803 11700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.731881 11700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:36.731925 11700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.740869 11700 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:46089
I20260924 16:12:36.740903 11827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:46089 every 8 connection(s)
I20260924 16:12:36.741744 11700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:36.746301 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 11700
I20260924 16:12:36.746423 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:36.746837 2037 external_mini_cluster.cc:1428] Setting key fa0f9343a232413e50d0948c3352dcb3
I20260924 16:12:36.751200 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:36.743529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:38609 (local address 127.1.253.126:44707)
0924 16:12:36.743666 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:36.743670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:36.744708 (+ 1038us) server_negotiation.cc:407] Connection header received
0924 16:12:36.746037 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:36.746042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:36.746115 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:36.746222 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:36.747314 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.747925 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.749219 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.749455 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.749566 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:36.750701 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:36.750705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:36.750708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:36.750721 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:36.750747 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:36.750755 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:36.750829 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:36.750999 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 16:12:36.751061 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:12:36.752038 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:36.743955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:44707 (local address 127.1.253.67:38609)
0924 16:12:36.744565 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:36.744604 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:36.745811 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:36.746455 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:36.746463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:36.746572 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:36.747167 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:36.747181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.748051 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.748055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:36.749080 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:36.749090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.749603 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.749607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:36.749780 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:36.750445 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:36.750447 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:36.750484 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:36.750546 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:36.750844 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:36.750853 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:36.750858 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:12:36.750985 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 16:12:36.751262 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":106,"threads_started":1}
I20260924 16:12:36.753116 11828 heartbeater.cc:347] Connected to a master server at 127.1.253.126:44707
I20260924 16:12:36.753352 11828 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:36.753839 11828 heartbeater.cc:511] Master 127.1.253.126:44707 requested a full tablet report, sending...
I20260924 16:12:36.754892 11373 ts_manager.cc:194] Registered new tserver with Master: c985c57ed7f84af99358d631a521bac7 (127.1.253.67:46089)
I20260924 16:12:36.755453 11373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.253.67:38609
I20260924 16:12:36.760666 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:12:36.763530 11417 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:36.762656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.1.253.126:44707)
0924 16:12:36.762784 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:36.762788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:36.762991 (+ 203us) server_negotiation.cc:407] Connection header received
0924 16:12:36.763078 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:36.763089 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:36.763127 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:36.763195 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:36.763430 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49438: BlockingRecv error: recv got EOF from 127.0.0.1:49438 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:12:36.764035 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 11438
I20260924 16:12:36.771842 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 11569
I20260924 16:12:36.779623 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 11700
I20260924 16:12:36.784891 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 11347
2026-09-24T16:12:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (897 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T16:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:36Z Disabled control of system clock
I20260924 16:12:36.814229 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:45009
--webserver_interface=127.1.253.126
--webserver_port=0
--builtin_ntp_servers=127.1.253.84:45153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:45009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 16:12:36.926977 11839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:36.927217 11839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:36.927253 11839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:36.931303 11839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:36.931352 11839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:36.931367 11839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:36.931384 11839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:36.931399 11839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:36.936116 11839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:45009
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:45009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:36.937134 11839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:36.938535 11839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:36.946897 11844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:36.946907 11845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:36.947134 11839 server_base.cc:1034] running on GCE node
W20260924 16:12:36.946907 11847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:36.947813 11839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:36.948765 11839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:36.949939 11839 hybrid_clock.cc:648] HybridClock initialized: now 1790266356949917 us; error 43 us; skew 500 ppm
I20260924 16:12:36.951720 11839 webserver.cc:460] Webserver started at http://127.1.253.126:33079/ using document root <none> and password file <none>
I20260924 16:12:36.952296 11839 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:36.952407 11839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:36.952657 11839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:36.954677 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "ef4a4120438540859318cddeb160f5bb"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "323a277b5fed644a673dfc0c9dbc2409"
server_key_iv: "3eb0e8ebfdc05a7456ec8306a96adf15"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.955255 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "ef4a4120438540859318cddeb160f5bb"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "323a277b5fed644a673dfc0c9dbc2409"
server_key_iv: "3eb0e8ebfdc05a7456ec8306a96adf15"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.958954 11839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:12:36.961338 11853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.962637 11839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 16:12:36.962785 11839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "ef4a4120438540859318cddeb160f5bb"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-6wvm"
server_key: "323a277b5fed644a673dfc0c9dbc2409"
server_key_iv: "3eb0e8ebfdc05a7456ec8306a96adf15"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.962903 11839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:36.967521 11839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:36.968147 11839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:36.968339 11839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:36.975167 11839 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:45009
I20260924 16:12:36.975224 11905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:45009 every 8 connection(s)
I20260924 16:12:36.976140 11839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:36.979368 11906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:36.980180 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 11839
I20260924 16:12:36.980316 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:36.980640 2037 external_mini_cluster.cc:1428] Setting key 18100d5175c74e604d17d626b7960e23
I20260924 16:12:36.985754 11906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb: Bootstrap starting.
I20260924 16:12:36.986403 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:36.982198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.1.253.126:45009)
0924 16:12:36.982848 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:36.982863 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:36.982898 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:12:36.983531 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:36.983548 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:36.983898 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:36.984231 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:36.985042 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:36.985047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:36.985059 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:36.985070 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:36.985126 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:36.985143 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:36.985306 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:36.985377 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 16:12:36.985615 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":123,"threads_started":1}
I20260924 16:12:36.988186 11906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:36.989002 11906 log.cc:824] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:36.990865 11906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb: No bootstrap required, opened a new log
I20260924 16:12:36.994385 11906 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a4120438540859318cddeb160f5bb" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45009 } }
I20260924 16:12:36.994617 11906 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:36.994702 11906 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4a4120438540859318cddeb160f5bb, State: Initialized, Role: FOLLOWER
I20260924 16:12:36.995198 11906 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a4120438540859318cddeb160f5bb" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45009 } }
I20260924 16:12:36.995347 11906 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:36.995434 11906 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:36.995532 11906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:36.996388 11906 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a4120438540859318cddeb160f5bb" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45009 } }
I20260924 16:12:36.996733 11906 leader_election.cc:302] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef4a4120438540859318cddeb160f5bb; no voters:
I20260924 16:12:36.997083 11906 leader_election.cc:288] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:36.997193 11911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:36.997433 11911 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [term 1 LEADER]: Becoming Leader. State: Replica: ef4a4120438540859318cddeb160f5bb, State: Running, Role: LEADER
I20260924 16:12:36.997761 11911 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a4120438540859318cddeb160f5bb" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45009 } }
I20260924 16:12:36.998354 11906 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:36.999435 11913 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef4a4120438540859318cddeb160f5bb. Latest consensus state: current_term: 1 leader_uuid: "ef4a4120438540859318cddeb160f5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a4120438540859318cddeb160f5bb" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45009 } } }
I20260924 16:12:36.999558 11913 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:36.999760 11912 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef4a4120438540859318cddeb160f5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a4120438540859318cddeb160f5bb" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45009 } } }
I20260924 16:12:36.999914 11912 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:36.999934 11920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:37.003175 11920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:37.008903 11920 catalog_manager.cc:1347] Generated new cluster ID: 70139c669d06456397f6c003820a3df3
I20260924 16:12:37.008982 11920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:37.017544 11920 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:37.017856 11920 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:37.022346 11920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef4a4120438540859318cddeb160f5bb: Generated new TSK 0
I20260924 16:12:37.023030 11920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:12:37.034315 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45009
--builtin_ntp_servers=127.1.253.84:45153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 16:12:37.164541 11930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:37.164814 11930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:37.164855 11930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:37.169215 11930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:37.169283 11930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:37.169376 11930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:12:37.174450 11930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:37.175647 11930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:37.177153 11930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:37.186932 11938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:37.186944 11936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:37.187037 11930 server_base.cc:1034] running on GCE node
W20260924 16:12:37.187233 11935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:37.187640 11930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:37.188268 11930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:37.189456 11930 hybrid_clock.cc:648] HybridClock initialized: now 1790266357189440 us; error 31 us; skew 500 ppm
I20260924 16:12:37.191347 11930 webserver.cc:460] Webserver started at http://127.1.253.65:35393/ using document root <none> and password file <none>
I20260924 16:12:37.191932 11930 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:37.192024 11930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:37.192262 11930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:37.194360 11930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "91ebd345f57140208ec4207a4e7066a4"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-6wvm"
server_key: "920ae610a8963ea73caf1da8b58b5921"
server_key_iv: "dbec3d216a4090bbbaf448abd535187b"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.194921 11930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "91ebd345f57140208ec4207a4e7066a4"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-6wvm"
server_key: "920ae610a8963ea73caf1da8b58b5921"
server_key_iv: "dbec3d216a4090bbbaf448abd535187b"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.198726 11930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 16:12:37.201531 11944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.203020 11930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:37.203155 11930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "91ebd345f57140208ec4207a4e7066a4"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-6wvm"
server_key: "920ae610a8963ea73caf1da8b58b5921"
server_key_iv: "dbec3d216a4090bbbaf448abd535187b"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.203276 11930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:37.207849 11930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:37.208483 11930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:37.208662 11930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:37.209203 11930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:37.210110 11930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:37.210182 11930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.210276 11930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:37.210330 11930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.219218 11930 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:38761
I20260924 16:12:37.219242 12057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:38761 every 8 connection(s)
I20260924 16:12:37.220149 11930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:37.221863 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 11930
I20260924 16:12:37.221988 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:37.222368 2037 external_mini_cluster.cc:1428] Setting key b820cc3a82bc148d168537829fa1730b
I20260924 16:12:37.224584 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45009
--builtin_ntp_servers=127.1.253.84:45153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 16:12:37.226884 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:37.222203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:47117 (local address 127.1.253.126:45009)
0924 16:12:37.222386 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:37.222390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:37.223682 (+ 1292us) server_negotiation.cc:407] Connection header received
0924 16:12:37.225055 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:37.225059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:37.225101 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:37.225178 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:37.226439 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:37.226443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:37.226445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:37.226457 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:37.226479 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:37.226487 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:37.226576 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:37.226739 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 16:12:37.226776 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 16:12:37.227905 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:37.222760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45009 (local address 127.1.253.65:47117)
0924 16:12:37.223511 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:37.223563 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:37.224844 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:37.225344 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:37.225351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:37.225439 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:37.226090 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:37.226092 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:37.226151 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:37.226271 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:37.226580 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:37.226592 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:37.226599 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:12:37.226787 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 16:12:37.227052 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":136,"threads_started":1}
I20260924 16:12:37.229192 12058 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45009
I20260924 16:12:37.229441 12058 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:37.229864 12058 heartbeater.cc:511] Master 127.1.253.126:45009 requested a full tablet report, sending...
I20260924 16:12:37.231454 11870 ts_manager.cc:194] Registered new tserver with Master: 91ebd345f57140208ec4207a4e7066a4 (127.1.253.65:38761)
W20260924 16:12:37.342772 12061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:37.343071 12061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:37.343152 12061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:37.347162 12061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:37.347237 12061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:37.347334 12061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:12:37.352344 12061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:37.353492 12061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:37.355073 12061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:37.364969 12067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:37.364964 12069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:37.365182 12061 server_base.cc:1034] running on GCE node
W20260924 16:12:37.365006 12066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:37.365691 12061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:37.366473 12061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:37.367653 12061 hybrid_clock.cc:648] HybridClock initialized: now 1790266357367636 us; error 27 us; skew 500 ppm
I20260924 16:12:37.369663 12061 webserver.cc:460] Webserver started at http://127.1.253.66:39641/ using document root <none> and password file <none>
I20260924 16:12:37.370347 12061 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:37.370421 12061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:37.370728 12061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:37.373078 12061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "0d6b1602b8ee44929a95abab725c06bf"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-6wvm"
server_key: "723ae4bdeb506521a159622c11e609ba"
server_key_iv: "71eaf310396b7923758809c3c3afbc80"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.373678 12061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "0d6b1602b8ee44929a95abab725c06bf"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-6wvm"
server_key: "723ae4bdeb506521a159622c11e609ba"
server_key_iv: "71eaf310396b7923758809c3c3afbc80"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.377620 12061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:12:37.380328 12075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.381664 12061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:37.381798 12061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "0d6b1602b8ee44929a95abab725c06bf"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-6wvm"
server_key: "723ae4bdeb506521a159622c11e609ba"
server_key_iv: "71eaf310396b7923758809c3c3afbc80"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.381917 12061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:37.386518 12061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:37.387135 12061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:37.387318 12061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:37.387843 12061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:37.388753 12061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:37.388824 12061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.388890 12061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:37.388942 12061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.397601 12061 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:45991
I20260924 16:12:37.397635 12188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:45991 every 8 connection(s)
I20260924 16:12:37.398566 12061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:37.401994 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 12061
I20260924 16:12:37.402134 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:37.402477 2037 external_mini_cluster.cc:1428] Setting key 5810ce97c17a4f0b8b7348063bcc2390
I20260924 16:12:37.404757 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45009
--builtin_ntp_servers=127.1.253.84:45153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 16:12:37.405261 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:37.400489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:43107 (local address 127.1.253.126:45009)
0924 16:12:37.400650 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:37.400655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:37.401763 (+ 1108us) server_negotiation.cc:407] Connection header received
0924 16:12:37.403296 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:37.403299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:37.403341 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:37.403415 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:37.404669 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:37.404673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:37.404675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:37.404688 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:37.404712 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:37.404723 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:37.404820 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:37.405059 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 16:12:37.405111 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:12:37.406239 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:37.400924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45009 (local address 127.1.253.66:43107)
0924 16:12:37.401599 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:37.401639 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:37.403086 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:37.403588 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:37.403597 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:37.403707 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:37.404386 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:37.404388 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:37.404441 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:37.404508 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:37.404812 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:37.404819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:37.404823 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:12:37.405081 (+ 258us) client_negotiation.cc:240] Negotiation successful
0924 16:12:37.405338 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260924 16:12:37.407485 12189 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45009
I20260924 16:12:37.407735 12189 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:37.408206 12189 heartbeater.cc:511] Master 127.1.253.126:45009 requested a full tablet report, sending...
I20260924 16:12:37.409322 11870 ts_manager.cc:194] Registered new tserver with Master: 0d6b1602b8ee44929a95abab725c06bf (127.1.253.66:45991)
W20260924 16:12:37.521605 12192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:37.522017 12192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:37.522132 12192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:37.530432 12192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:37.530506 12192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:37.530627 12192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:12:37.538882 12192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:37.540308 12192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:37.541926 12192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:37.553398 12197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:37.553604 12198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:37.553434 12192 server_base.cc:1034] running on GCE node
W20260924 16:12:37.553390 12200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:37.554227 12192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:37.555014 12192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:37.556200 12192 hybrid_clock.cc:648] HybridClock initialized: now 1790266357556172 us; error 42 us; skew 500 ppm
I20260924 16:12:37.558352 12192 webserver.cc:460] Webserver started at http://127.1.253.67:41895/ using document root <none> and password file <none>
I20260924 16:12:37.559096 12192 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:37.559196 12192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:37.559429 12192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:37.562222 12192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "c12b85cab4ac45659be6e5d7e472ed89"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-6wvm"
server_key: "c8d8571699f22429a03ecd01166559ab"
server_key_iv: "17ed6782ac5d06d55a8a3595bd25c77b"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.562784 12192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "c12b85cab4ac45659be6e5d7e472ed89"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-6wvm"
server_key: "c8d8571699f22429a03ecd01166559ab"
server_key_iv: "17ed6782ac5d06d55a8a3595bd25c77b"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.566502 12192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:12:37.569027 12206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.570395 12192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:37.570494 12192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "c12b85cab4ac45659be6e5d7e472ed89"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-6wvm"
server_key: "c8d8571699f22429a03ecd01166559ab"
server_key_iv: "17ed6782ac5d06d55a8a3595bd25c77b"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.570581 12192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:37.575006 12192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:37.575573 12192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:37.575706 12192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:37.576200 12192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:37.577065 12192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:37.577111 12192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.577150 12192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:37.577165 12192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.586555 12192 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:39439
I20260924 16:12:37.586572 12319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:39439 every 8 connection(s)
I20260924 16:12:37.587507 12192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:37.591958 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 12192
I20260924 16:12:37.592082 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:37.592434 2037 external_mini_cluster.cc:1428] Setting key e2f27d3cb3d80e038a14e72b3c4f7381
I20260924 16:12:37.594122 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:37.589340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:44359 (local address 127.1.253.126:45009)
0924 16:12:37.589524 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:37.589529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:37.590684 (+ 1155us) server_negotiation.cc:407] Connection header received
0924 16:12:37.592096 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:37.592100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:37.592157 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:37.592235 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:37.593603 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:37.593606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:37.593609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:37.593620 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:37.593640 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:37.593648 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:37.593763 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:37.593926 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 16:12:37.593990 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:12:37.595094 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:37.589800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45009 (local address 127.1.253.67:44359)
0924 16:12:37.590531 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:37.590572 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:37.591836 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:37.592456 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:37.592468 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:37.592584 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:37.593263 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:37.593265 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:37.593341 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:37.593456 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:37.593760 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:37.593767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:37.593771 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:12:37.593961 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 16:12:37.594192 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
I20260924 16:12:37.596145 12320 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45009
I20260924 16:12:37.596376 12320 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:37.596784 12320 heartbeater.cc:511] Master 127.1.253.126:45009 requested a full tablet report, sending...
I20260924 16:12:37.597761 11870 ts_manager.cc:194] Registered new tserver with Master: c12b85cab4ac45659be6e5d7e472ed89 (127.1.253.67:39439)
I20260924 16:12:37.606587 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:12:37.609341 11909 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:37.608571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.1.253.126:45009)
0924 16:12:37.608704 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:37.608707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:37.608874 (+ 167us) server_negotiation.cc:407] Connection header received
0924 16:12:37.608975 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:37.608977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:37.609012 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:37.609077 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:37.609241 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49992: BlockingRecv error: recv got EOF from 127.0.0.1:49992 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:12:37.609541 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 11930
I20260924 16:12:37.616462 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 12061
I20260924 16:12:37.622131 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 12192
I20260924 16:12:37.627489 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 11839
2026-09-24T16:12:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (842 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:37 dist-test-slave-6wvm krb5kdc[12330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:37 dist-test-slave-6wvm krb5kdc[12330](info): set up 2 sockets
Sep 24 16:12:37 dist-test-slave-6wvm krb5kdc[12330](info): commencing operation
krb5kdc: starting...
W20260924 16:12:38.671116 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:12:38 dist-test-slave-6wvm krb5kdc[12330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:12:38.792879 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:43967
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:43967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:38.907634 12343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:38.907955 12343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:38.908041 12343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:38.912177 12343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:38.912281 12343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:38.912324 12343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:38.912385 12343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:38.912438 12343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:38.917860 12343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:43967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:43967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:38.919098 12343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:38.920246 12343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:38.928694 12349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:38.928826 12343 server_base.cc:1034] running on GCE node
W20260924 16:12:38.928736 12351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:38.928732 12348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:38.929512 12343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:38.930747 12343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:38.931957 12343 hybrid_clock.cc:648] HybridClock initialized: now 1790266358931921 us; error 45 us; skew 500 ppm
Sep 24 16:12:38 dist-test-slave-6wvm krb5kdc[12330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:38.934788 12343 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:12:38.935657 12343 webserver.cc:460] Webserver started at http://127.1.253.126:36281/ using document root <none> and password file <none>
I20260924 16:12:38.936208 12343 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:38.936298 12343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:38.936548 12343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:38.938686 12343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "939d60a6a8884438bf42dcdb411d22eb"
format_stamp: "Formatted at 2026-09-24 16:12:38 on dist-test-slave-6wvm"
server_key: "5193f19776aadc0817adf463a32c981f"
server_key_iv: "f7eba13a313018be2ccb73456e604fa7"
server_key_version: "encryptionkey@0"
I20260924 16:12:38.939247 12343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "939d60a6a8884438bf42dcdb411d22eb"
format_stamp: "Formatted at 2026-09-24 16:12:38 on dist-test-slave-6wvm"
server_key: "5193f19776aadc0817adf463a32c981f"
server_key_iv: "f7eba13a313018be2ccb73456e604fa7"
server_key_version: "encryptionkey@0"
I20260924 16:12:38.942983 12343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:38.945446 12358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:38.946686 12343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:38.946832 12343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "939d60a6a8884438bf42dcdb411d22eb"
format_stamp: "Formatted at 2026-09-24 16:12:38 on dist-test-slave-6wvm"
server_key: "5193f19776aadc0817adf463a32c981f"
server_key_iv: "f7eba13a313018be2ccb73456e604fa7"
server_key_version: "encryptionkey@0"
I20260924 16:12:38.946936 12343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:38.956959 12343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:38.960073 12343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:38.960294 12343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:38.967258 12343 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:43967
I20260924 16:12:38.967558 12410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:43967 every 8 connection(s)
I20260924 16:12:38.968318 12343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:38.969254 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 12343
I20260924 16:12:38.969383 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:12:38.969671 2037 external_mini_cluster.cc:1428] Setting key 7bb9dbbd5c80f6223d87de498906b235
I20260924 16:12:38.971643 12411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:38.978570 12411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb: Bootstrap starting.
Sep 24 16:12:38 dist-test-slave-6wvm krb5kdc[12330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:38.981320 12411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:38.982280 12411 log.cc:824] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:38.984570 12411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb: No bootstrap required, opened a new log
I20260924 16:12:38.985227 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:38.971196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.1.253.126:43967)
0924 16:12:38.971643 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:38.971674 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:38.971720 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:12:38.972546 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:38.972569 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:38.972993 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:38.973357 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:38.974026 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.975585 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.976434 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.977127 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.977293 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:38.979875 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:38.979881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:38.979898 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:38.979929 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:38.981924 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.982400 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.982406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.982412 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.982552 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.982851 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.982854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.982856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.983359 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:38.983511 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:38.983866 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 16:12:38.984240 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":115,"threads_started":1}
I20260924 16:12:38.987738 12411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939d60a6a8884438bf42dcdb411d22eb" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43967 } }
I20260924 16:12:38.987953 12411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:38.987987 12411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939d60a6a8884438bf42dcdb411d22eb, State: Initialized, Role: FOLLOWER
I20260924 16:12:38.988430 12411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939d60a6a8884438bf42dcdb411d22eb" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43967 } }
I20260924 16:12:38.988575 12411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:38.988639 12411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:38.988732 12411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:38.989624 12411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939d60a6a8884438bf42dcdb411d22eb" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43967 } }
I20260924 16:12:38.989977 12411 leader_election.cc:302] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 939d60a6a8884438bf42dcdb411d22eb; no voters:
I20260924 16:12:38.990334 12411 leader_election.cc:288] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:38.990466 12416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:38.990708 12416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [term 1 LEADER]: Becoming Leader. State: Replica: 939d60a6a8884438bf42dcdb411d22eb, State: Running, Role: LEADER
I20260924 16:12:38.991056 12416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939d60a6a8884438bf42dcdb411d22eb" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43967 } }
I20260924 16:12:38.991536 12411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:38.992974 12418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 939d60a6a8884438bf42dcdb411d22eb. Latest consensus state: current_term: 1 leader_uuid: "939d60a6a8884438bf42dcdb411d22eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939d60a6a8884438bf42dcdb411d22eb" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43967 } } }
I20260924 16:12:38.993027 12417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "939d60a6a8884438bf42dcdb411d22eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939d60a6a8884438bf42dcdb411d22eb" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43967 } } }
I20260924 16:12:38.993094 12418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:38.993124 12417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:38.993954 12426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:38.996562 12426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:39.002146 12426 catalog_manager.cc:1347] Generated new cluster ID: 08f33a80f4834251a3e204f41f178dbc
I20260924 16:12:39.002238 12426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:39.013703 12426 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:39.014760 12426 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:39.023046 12426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb: Generated new TSK 0
I20260924 16:12:39.023828 12426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:12:39.059617 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43967
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:39.174330 12437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:39.174636 12437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:39.174713 12437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:39.178725 12437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:39.178817 12437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:39.178911 12437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:12:39.184154 12437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:39.185292 12437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:39.186414 12437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:39.196399 12442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:39.196424 12445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:39.196538 12437 server_base.cc:1034] running on GCE node
W20260924 16:12:39.196698 12443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:39.197177 12437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:39.197835 12437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:39.198997 12437 hybrid_clock.cc:648] HybridClock initialized: now 1790266359198977 us; error 30 us; skew 500 ppm
Sep 24 16:12:39 dist-test-slave-6wvm krb5kdc[12330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:39.201749 12437 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:12:39.202716 12437 webserver.cc:460] Webserver started at http://127.1.253.65:46535/ using document root <none> and password file <none>
I20260924 16:12:39.203301 12437 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:39.203363 12437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:39.203608 12437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:39.205729 12437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "269fe5a1dbc04f858fc966095df72232"
format_stamp: "Formatted at 2026-09-24 16:12:39 on dist-test-slave-6wvm"
server_key: "9e4b02a8ad8896e21bd76a8487b14d2e"
server_key_iv: "04cc7cb521ab86d4542403b7bc3d90f2"
server_key_version: "encryptionkey@0"
I20260924 16:12:39.206394 12437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "269fe5a1dbc04f858fc966095df72232"
format_stamp: "Formatted at 2026-09-24 16:12:39 on dist-test-slave-6wvm"
server_key: "9e4b02a8ad8896e21bd76a8487b14d2e"
server_key_iv: "04cc7cb521ab86d4542403b7bc3d90f2"
server_key_version: "encryptionkey@0"
I20260924 16:12:39.210144 12437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:39.212790 12452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:39.213946 12437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:39.214092 12437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "269fe5a1dbc04f858fc966095df72232"
format_stamp: "Formatted at 2026-09-24 16:12:39 on dist-test-slave-6wvm"
server_key: "9e4b02a8ad8896e21bd76a8487b14d2e"
server_key_iv: "04cc7cb521ab86d4542403b7bc3d90f2"
server_key_version: "encryptionkey@0"
I20260924 16:12:39.214201 12437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:39.245318 12437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:39.248387 12437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:39.248598 12437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:39.249220 12437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:39.250064 12437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:39.250136 12437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:39.250204 12437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:39.250272 12437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:39.259186 12437 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:40807
I20260924 16:12:39.259208 12565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:40807 every 8 connection(s)
I20260924 16:12:39.260094 12437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:39.266954 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 12437
I20260924 16:12:39.267084 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:39.267382 2037 external_mini_cluster.cc:1428] Setting key b461288287a2bcc831fd40aead9b6704
Sep 24 16:12:39 dist-test-slave-6wvm krb5kdc[12330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:39.275043 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.261886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:38369 (local address 127.1.253.126:43967)
0924 16:12:39.262056 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:39.262062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:39.262971 (+ 909us) server_negotiation.cc:407] Connection header received
0924 16:12:39.264313 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:39.264316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:39.264367 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:39.264448 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:39.265646 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.266178 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.267862 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.268229 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.268387 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.271360 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:39.271363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:39.271365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:39.271389 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:39.272921 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:39.273463 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:39.273468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:39.273470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:39.273555 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:39.273912 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:39.273916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:39.273918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:39.274106 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:39.274217 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:39.274737 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 16:12:39.274879 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:12:39.275887 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.262353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:43967 (local address 127.1.253.65:38369)
0924 16:12:39.262809 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:39.262853 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:39.264084 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:39.264618 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:39.264625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:39.265010 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:39.265491 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.265506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.266363 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.266367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.267662 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.267675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.268431 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.268436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.268642 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.269312 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:39.269337 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:39.271178 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:39.273050 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:39.273057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:39.273073 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:39.273334 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:39.273649 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:39.273653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:39.273655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:39.273812 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:39.274216 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:39.274222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:39.274498 (+ 276us) client_negotiation.cc:769] Sending connection context
0924 16:12:39.274696 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 16:12:39.275048 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
I20260924 16:12:39.277199 12566 heartbeater.cc:347] Connected to a master server at 127.1.253.126:43967
I20260924 16:12:39.277449 12566 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:39.277925 12566 heartbeater.cc:511] Master 127.1.253.126:43967 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:12:39.279641 12375 ts_manager.cc:194] Registered new tserver with Master: 269fe5a1dbc04f858fc966095df72232 (127.1.253.65:40807)
I20260924 16:12:39.281003 12375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:38369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:12:39.296594 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43967
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:39.407094 12571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:39.407403 12571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:39.407480 12571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:39.411459 12571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:39.411558 12571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:39.411651 12571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:12:39.416828 12571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:39.418023 12571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:39.419162 12571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:39.428921 12576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:39.428977 12577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:39.428970 12571 server_base.cc:1034] running on GCE node
W20260924 16:12:39.429026 12579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:39.429534 12571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:39.430290 12571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:39.431483 12571 hybrid_clock.cc:648] HybridClock initialized: now 1790266359431462 us; error 31 us; skew 500 ppm
Sep 24 16:12:39 dist-test-slave-6wvm krb5kdc[12330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:39.434094 12571 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:12:39.434962 12571 webserver.cc:460] Webserver started at http://127.1.253.66:45811/ using document root <none> and password file <none>
I20260924 16:12:39.435532 12571 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:39.435644 12571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:39.435878 12571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:39.437865 12571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "fcac3d8c040d49fb8a4aa34a0728264e"
format_stamp: "Formatted at 2026-09-24 16:12:39 on dist-test-slave-6wvm"
server_key: "f437b86767dd32ebe3df6848608ef8fc"
server_key_iv: "d80b7365b0eec00c5fe4aab69a58cfa6"
server_key_version: "encryptionkey@0"
I20260924 16:12:39.438469 12571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "fcac3d8c040d49fb8a4aa34a0728264e"
format_stamp: "Formatted at 2026-09-24 16:12:39 on dist-test-slave-6wvm"
server_key: "f437b86767dd32ebe3df6848608ef8fc"
server_key_iv: "d80b7365b0eec00c5fe4aab69a58cfa6"
server_key_version: "encryptionkey@0"
I20260924 16:12:39.442129 12571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:39.444589 12586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:39.445864 12571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:12:39.446002 12571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "fcac3d8c040d49fb8a4aa34a0728264e"
format_stamp: "Formatted at 2026-09-24 16:12:39 on dist-test-slave-6wvm"
server_key: "f437b86767dd32ebe3df6848608ef8fc"
server_key_iv: "d80b7365b0eec00c5fe4aab69a58cfa6"
server_key_version: "encryptionkey@0"
I20260924 16:12:39.446126 12571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:39.461467 12571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:39.464480 12571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:39.464687 12571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:39.465292 12571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:39.466150 12571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:39.466220 12571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:39.466315 12571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:39.466361 12571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:39.475232 12571 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:35517
I20260924 16:12:39.475255 12699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:35517 every 8 connection(s)
I20260924 16:12:39.476186 12571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:39.483084 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 12571
I20260924 16:12:39.483197 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:39.483537 2037 external_mini_cluster.cc:1428] Setting key de1d924d4df718c1c9f542624aa4d2d6
Sep 24 16:12:39 dist-test-slave-6wvm krb5kdc[12330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:39.492008 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.478078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:40097 (local address 127.1.253.126:43967)
0924 16:12:39.478292 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:39.478309 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:39.479165 (+ 856us) server_negotiation.cc:407] Connection header received
0924 16:12:39.480519 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:39.480522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:39.480570 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:39.480645 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:39.481792 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.482512 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.484205 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.484572 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.484707 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.487897 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:39.487900 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:39.487902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:39.487925 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:39.489271 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:39.490023 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:39.490028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:39.490032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:39.490152 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:39.490658 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:39.490661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:39.490662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:39.490873 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:39.490999 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:39.491683 (+ 684us) server_negotiation.cc:299] Negotiation successful
0924 16:12:39.491804 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:12:39.493242 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.478513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:43967 (local address 127.1.253.66:40097)
0924 16:12:39.478997 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:39.479051 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:39.480316 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:39.480810 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:39.480817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:39.481183 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:39.481664 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.481678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.482683 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.482687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.484070 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.484086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.484757 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.484761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.485120 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.485778 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:39.485801 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:39.487706 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:39.489391 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:39.489400 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:39.489417 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:39.489845 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:39.490350 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:39.490353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:39.490355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:39.490548 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:39.491000 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:39.491010 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:39.491379 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 16:12:39.491620 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 16:12:39.492020 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:12:39.494593 12700 heartbeater.cc:347] Connected to a master server at 127.1.253.126:43967
I20260924 16:12:39.494882 12700 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:39.495427 12700 heartbeater.cc:511] Master 127.1.253.126:43967 requested a full tablet report, sending...
I20260924 16:12:39.496692 12375 ts_manager.cc:194] Registered new tserver with Master: fcac3d8c040d49fb8a4aa34a0728264e (127.1.253.66:35517)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
I20260924 16:12:39.497537 12375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:40097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:12:39.512652 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43967
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:39.624634 12705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:39.624929 12705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:39.625010 12705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:39.628922 12705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:39.629004 12705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:39.629092 12705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:12:39.634047 12705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:12:39.635092 12705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:39.636111 12705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:39.645695 12713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:39.645706 12711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:39.645773 12705 server_base.cc:1034] running on GCE node
W20260924 16:12:39.645720 12710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:39.646471 12705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:39.647078 12705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:39.648247 12705 hybrid_clock.cc:648] HybridClock initialized: now 1790266359648232 us; error 25 us; skew 500 ppm
Sep 24 16:12:39 dist-test-slave-6wvm krb5kdc[12330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:39.654829 12705 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:12:39.655735 12705 webserver.cc:460] Webserver started at http://127.1.253.67:41653/ using document root <none> and password file <none>
I20260924 16:12:39.656317 12705 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:39.656378 12705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:39.656616 12705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:39.658799 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "272cd765be064903b64769103b8ccfc5"
format_stamp: "Formatted at 2026-09-24 16:12:39 on dist-test-slave-6wvm"
server_key: "a04ed4ab6c6d5ceba4a612f235fe4fbf"
server_key_iv: "e4fcceec3765d7be0d190c36e0574b87"
server_key_version: "encryptionkey@0"
I20260924 16:12:39.659379 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "272cd765be064903b64769103b8ccfc5"
format_stamp: "Formatted at 2026-09-24 16:12:39 on dist-test-slave-6wvm"
server_key: "a04ed4ab6c6d5ceba4a612f235fe4fbf"
server_key_iv: "e4fcceec3765d7be0d190c36e0574b87"
server_key_version: "encryptionkey@0"
I20260924 16:12:39.663312 12705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:12:39.665774 12720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:39.667093 12705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:12:39.667237 12705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "272cd765be064903b64769103b8ccfc5"
format_stamp: "Formatted at 2026-09-24 16:12:39 on dist-test-slave-6wvm"
server_key: "a04ed4ab6c6d5ceba4a612f235fe4fbf"
server_key_iv: "e4fcceec3765d7be0d190c36e0574b87"
server_key_version: "encryptionkey@0"
I20260924 16:12:39.667354 12705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:39.706012 12705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:39.709265 12705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:39.709483 12705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:39.710141 12705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:39.711099 12705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:39.711172 12705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:39.711242 12705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:39.711290 12705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:39.721457 12705 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:33315
I20260924 16:12:39.721475 12834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:33315 every 8 connection(s)
I20260924 16:12:39.722486 12705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:39.731019 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 12705
I20260924 16:12:39.731171 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:39.731487 2037 external_mini_cluster.cc:1428] Setting key 8a64fe81464776c18e8c38d81fd46595
Sep 24 16:12:39 dist-test-slave-6wvm krb5kdc[12330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:12:39.736470 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.724395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:40183 (local address 127.1.253.126:43967)
0924 16:12:39.724567 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:39.724572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:39.725494 (+ 922us) server_negotiation.cc:407] Connection header received
0924 16:12:39.727009 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:39.727013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:39.727071 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:39.727161 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:39.728334 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.728853 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.729932 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.730135 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.730210 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.732939 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:39.732941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:39.732943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:39.732965 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:39.734434 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:39.734954 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:39.734959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:39.734961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:39.735040 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:39.735394 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:39.735398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:39.735399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:39.735582 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:39.735687 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:39.736183 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 16:12:39.736302 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 16:12:39.737319 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.724861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:43967 (local address 127.1.253.67:40183)
0924 16:12:39.725334 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:39.725380 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:39.726771 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:39.727343 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:39.727350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:39.727710 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:39.728190 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.728203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.728957 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.728959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.729830 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.729839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.730324 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.730329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.730482 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.731109 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:39.731124 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:39.732774 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:39.734546 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:39.734552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:39.734567 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:39.734847 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:39.735140 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:39.735143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:39.735146 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:39.735296 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:39.735696 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:39.735704 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:39.735951 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 16:12:39.736142 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 16:12:39.736473 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":121,"threads_started":1}
I20260924 16:12:39.738369 12835 heartbeater.cc:347] Connected to a master server at 127.1.253.126:43967
I20260924 16:12:39.738620 12835 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:39.739099 12835 heartbeater.cc:511] Master 127.1.253.126:43967 requested a full tablet report, sending...
I20260924 16:12:39.740149 12375 ts_manager.cc:194] Registered new tserver with Master: 272cd765be064903b64769103b8ccfc5 (127.1.253.67:33315)
I20260924 16:12:39.740715 12375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:40183
I20260924 16:12:39.745194 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:12:39.754545 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.747137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54828 (local address 127.1.253.126:43967)
0924 16:12:39.747277 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:39.747281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:39.747483 (+ 202us) server_negotiation.cc:407] Connection header received
0924 16:12:39.747584 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:39.747587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:39.747629 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:39.747689 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:39.748408 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.748966 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.749836 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.750112 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.750221 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.751686 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:39.751689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:39.751692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:39.751717 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:39.752910 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:39.753414 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:39.753417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:39.753418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:39.753501 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:39.753740 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:39.753743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:39.753744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:39.753952 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:39.754054 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:39.754325 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 16:12:39.754425 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:12:39.757491 12375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:39.760532 12375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:39.774904 12846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.769039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:35517 (local address 127.0.0.1:36486)
0924 16:12:39.769847 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:39.769897 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:39.770034 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:39.770818 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:39.770824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:39.770850 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:39.770982 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.770991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.772248 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.772253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.773487 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.773497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.774334 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.774337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.774439 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.774474 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:12:39.774629 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 16:12:39.774757 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":35,"thread_start_us":89,"threads_started":1}
I20260924 16:12:39.774914 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.769194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.1.253.66:35517)
0924 16:12:39.770422 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:39.770434 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:39.770454 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:12:39.770518 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:39.770522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:39.770661 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:39.770793 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:39.771119 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.772113 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.773610 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.774169 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.774328 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.774483 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:39.774664 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 16:12:39.774757 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1123,"thread_start_us":158,"threads_started":1}
I20260924 16:12:39.776152 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.769779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:33315 (local address 127.0.0.1:49386)
0924 16:12:39.770417 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:39.770431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:39.770528 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:39.771133 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:39.771135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:39.771152 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:39.771309 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.771320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.772728 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.772731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.774208 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.774220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.775677 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.775681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.775823 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.775872 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 16:12:39.775946 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 16:12:39.776020 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":222,"thread_start_us":92,"threads_started":1}
I20260924 16:12:39.776485 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.769540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.1.253.65:40807)
0924 16:12:39.769912 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:39.769934 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:39.770159 (+ 225us) server_negotiation.cc:407] Connection header received
0924 16:12:39.770408 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:39.770416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:39.770646 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:39.770785 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:39.771359 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.772562 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.774375 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.775422 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.775490 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.775609 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:39.776010 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 16:12:39.776101 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":89,"threads_started":1}
I20260924 16:12:39.776902 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.769197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40807 (local address 127.0.0.1:33478)
0924 16:12:39.770059 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:39.770073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:39.770229 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:39.771037 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:39.771039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:39.771079 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:39.771230 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.771237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.772727 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.772731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.774208 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.774220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.775697 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.775700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.775850 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.775885 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:12:39.776677 (+ 792us) client_negotiation.cc:240] Negotiation successful
0924 16:12:39.776765 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":202,"thread_start_us":130,"threads_started":1}
I20260924 16:12:39.776913 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.770185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49386 (local address 127.1.253.67:33315)
0924 16:12:39.770657 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:39.770670 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:39.770694 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:12:39.770787 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:39.770792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:39.770981 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:39.771128 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:39.771449 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.772590 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.774375 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.775350 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.775469 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.775609 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:39.776421 (+ 812us) server_negotiation.cc:299] Negotiation successful
0924 16:12:39.776527 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":163,"threads_started":1}
I20260924 16:12:39.778051 12634 tablet_service.cc:1467] Processing CreateTablet for tablet 1821287a13b342c987064d519e21346d (DEFAULT_TABLE table=test-table [id=8a564ed8055646b984a5ffa0ca900251]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:39.779134 12634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1821287a13b342c987064d519e21346d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:39.779385 12500 tablet_service.cc:1467] Processing CreateTablet for tablet 1821287a13b342c987064d519e21346d (DEFAULT_TABLE table=test-table [id=8a564ed8055646b984a5ffa0ca900251]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:39.780412 12500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1821287a13b342c987064d519e21346d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:39.785284 12769 tablet_service.cc:1467] Processing CreateTablet for tablet 1821287a13b342c987064d519e21346d (DEFAULT_TABLE table=test-table [id=8a564ed8055646b984a5ffa0ca900251]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:39.785820 12852 tablet_bootstrap.cc:492] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e: Bootstrap starting.
I20260924 16:12:39.786418 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1821287a13b342c987064d519e21346d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:39.787992 12853 tablet_bootstrap.cc:492] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232: Bootstrap starting.
I20260924 16:12:39.788439 12852 tablet_bootstrap.cc:654] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:39.789103 12852 log.cc:824] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:39.790211 12853 tablet_bootstrap.cc:654] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:39.791096 12853 log.cc:824] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:39.791708 12852 tablet_bootstrap.cc:492] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e: No bootstrap required, opened a new log
I20260924 16:12:39.791944 12852 ts_tablet_manager.cc:1397] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:12:39.793184 12853 tablet_bootstrap.cc:492] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232: No bootstrap required, opened a new log
I20260924 16:12:39.793385 12853 ts_tablet_manager.cc:1397] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:12:39.793840 12856 tablet_bootstrap.cc:492] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5: Bootstrap starting.
I20260924 16:12:39.795929 12852 raft_consensus.cc:348] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcac3d8c040d49fb8a4aa34a0728264e" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35517 } } peers { permanent_uuid: "269fe5a1dbc04f858fc966095df72232" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "272cd765be064903b64769103b8ccfc5" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33315 } }
I20260924 16:12:39.796236 12852 raft_consensus.cc:374] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:39.796291 12852 raft_consensus.cc:729] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcac3d8c040d49fb8a4aa34a0728264e, State: Initialized, Role: FOLLOWER
I20260924 16:12:39.796677 12856 tablet_bootstrap.cc:654] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:39.796828 12852 consensus_queue.cc:260] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcac3d8c040d49fb8a4aa34a0728264e" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35517 } } peers { permanent_uuid: "269fe5a1dbc04f858fc966095df72232" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "272cd765be064903b64769103b8ccfc5" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33315 } }
I20260924 16:12:39.797065 12853 raft_consensus.cc:348] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcac3d8c040d49fb8a4aa34a0728264e" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35517 } } peers { permanent_uuid: "269fe5a1dbc04f858fc966095df72232" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "272cd765be064903b64769103b8ccfc5" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33315 } }
I20260924 16:12:39.797351 12853 raft_consensus.cc:374] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:39.797415 12853 raft_consensus.cc:729] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269fe5a1dbc04f858fc966095df72232, State: Initialized, Role: FOLLOWER
I20260924 16:12:39.797717 12700 heartbeater.cc:503] Master 127.1.253.126:43967 was elected leader, sending a full tablet report...
I20260924 16:12:39.797871 12853 consensus_queue.cc:260] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcac3d8c040d49fb8a4aa34a0728264e" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35517 } } peers { permanent_uuid: "269fe5a1dbc04f858fc966095df72232" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "272cd765be064903b64769103b8ccfc5" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33315 } }
I20260924 16:12:39.798115 12856 log.cc:824] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:39.798516 12852 ts_tablet_manager.cc:1428] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:12:39.798887 12853 ts_tablet_manager.cc:1428] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:12:39.799216 12566 heartbeater.cc:503] Master 127.1.253.126:43967 was elected leader, sending a full tablet report...
I20260924 16:12:39.800571 12856 tablet_bootstrap.cc:492] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5: No bootstrap required, opened a new log
I20260924 16:12:39.800799 12856 ts_tablet_manager.cc:1397] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 16:12:39.804126 12856 raft_consensus.cc:348] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcac3d8c040d49fb8a4aa34a0728264e" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35517 } } peers { permanent_uuid: "269fe5a1dbc04f858fc966095df72232" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "272cd765be064903b64769103b8ccfc5" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33315 } }
I20260924 16:12:39.804348 12856 raft_consensus.cc:374] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:39.804425 12856 raft_consensus.cc:729] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272cd765be064903b64769103b8ccfc5, State: Initialized, Role: FOLLOWER
I20260924 16:12:39.804879 12856 consensus_queue.cc:260] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcac3d8c040d49fb8a4aa34a0728264e" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35517 } } peers { permanent_uuid: "269fe5a1dbc04f858fc966095df72232" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "272cd765be064903b64769103b8ccfc5" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33315 } }
I20260924 16:12:39.805544 12835 heartbeater.cc:503] Master 127.1.253.126:43967 was elected leader, sending a full tablet report...
I20260924 16:12:39.805897 12856 ts_tablet_manager.cc:1428] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 16:12:39.862437 12858 raft_consensus.cc:482] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:39.862692 12858 raft_consensus.cc:504] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcac3d8c040d49fb8a4aa34a0728264e" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35517 } } peers { permanent_uuid: "269fe5a1dbc04f858fc966095df72232" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "272cd765be064903b64769103b8ccfc5" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33315 } }
I20260924 16:12:39.863912 12858 leader_election.cc:288] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcac3d8c040d49fb8a4aa34a0728264e (127.1.253.66:35517), 272cd765be064903b64769103b8ccfc5 (127.1.253.67:33315)
I20260924 16:12:39.868351 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.864286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:55415 (local address 127.1.253.66:35517)
0924 16:12:39.864430 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:39.864435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:39.864815 (+ 380us) server_negotiation.cc:407] Connection header received
0924 16:12:39.864999 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:39.865005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:39.865067 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:39.865152 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:39.865516 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.866056 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.867329 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.867725 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.867840 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.867902 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:39.868125 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 16:12:39.868202 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:12:39.868361 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.864458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:38917 (local address 127.1.253.67:33315)
0924 16:12:39.864590 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:39.864594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:39.864838 (+ 244us) server_negotiation.cc:407] Connection header received
0924 16:12:39.865078 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:39.865081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:39.865141 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:39.865218 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:39.865600 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.866084 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.867333 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.867742 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.867880 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.867933 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:39.868171 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 16:12:39.868252 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:12:39.868378 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.864235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:33315 (local address 127.1.253.65:38917)
0924 16:12:39.864751 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:39.864766 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:39.864939 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:39.865297 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:39.865300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:39.865318 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:39.865471 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.865481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.866291 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.866294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.867196 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.867206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.867893 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.867896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.867999 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.868016 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 16:12:39.868135 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 16:12:39.868200 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":247,"thread_start_us":109,"threads_started":1}
I20260924 16:12:39.868381 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.864232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:35517 (local address 127.1.253.65:55415)
0924 16:12:39.864707 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:39.864724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:39.864866 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:39.865180 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:39.865186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:39.865208 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:39.865379 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.865388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.866193 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.866199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.867191 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:39.867201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.867855 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.867859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:39.867953 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.867979 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 16:12:39.868073 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 16:12:39.868162 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":54,"thread_start_us":121,"threads_started":1}
I20260924 16:12:39.869127 12789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1821287a13b342c987064d519e21346d" candidate_uuid: "269fe5a1dbc04f858fc966095df72232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272cd765be064903b64769103b8ccfc5" is_pre_election: true
I20260924 16:12:39.869184 12654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1821287a13b342c987064d519e21346d" candidate_uuid: "269fe5a1dbc04f858fc966095df72232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcac3d8c040d49fb8a4aa34a0728264e" is_pre_election: true
I20260924 16:12:39.869491 12789 raft_consensus.cc:2454] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269fe5a1dbc04f858fc966095df72232 in term 0.
I20260924 16:12:39.869522 12654 raft_consensus.cc:2454] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269fe5a1dbc04f858fc966095df72232 in term 0.
I20260924 16:12:39.870015 12453 leader_election.cc:302] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 269fe5a1dbc04f858fc966095df72232, 272cd765be064903b64769103b8ccfc5, fcac3d8c040d49fb8a4aa34a0728264e; no voters:
I20260924 16:12:39.870329 12858 raft_consensus.cc:2781] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:39.870446 12858 raft_consensus.cc:482] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:39.870509 12858 raft_consensus.cc:3037] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:39.871598 12858 raft_consensus.cc:504] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcac3d8c040d49fb8a4aa34a0728264e" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35517 } } peers { permanent_uuid: "269fe5a1dbc04f858fc966095df72232" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "272cd765be064903b64769103b8ccfc5" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33315 } }
I20260924 16:12:39.871980 12858 leader_election.cc:288] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [CANDIDATE]: Term 1 election: Requested vote from peers fcac3d8c040d49fb8a4aa34a0728264e (127.1.253.66:35517), 272cd765be064903b64769103b8ccfc5 (127.1.253.67:33315)
I20260924 16:12:39.872347 12789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1821287a13b342c987064d519e21346d" candidate_uuid: "269fe5a1dbc04f858fc966095df72232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "272cd765be064903b64769103b8ccfc5"
I20260924 16:12:39.872430 12654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1821287a13b342c987064d519e21346d" candidate_uuid: "269fe5a1dbc04f858fc966095df72232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcac3d8c040d49fb8a4aa34a0728264e"
I20260924 16:12:39.872473 12789 raft_consensus.cc:3037] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:39.872545 12654 raft_consensus.cc:3037] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:39.873821 12654 raft_consensus.cc:2454] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269fe5a1dbc04f858fc966095df72232 in term 1.
I20260924 16:12:39.873821 12789 raft_consensus.cc:2454] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269fe5a1dbc04f858fc966095df72232 in term 1.
I20260924 16:12:39.874106 12453 leader_election.cc:302] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269fe5a1dbc04f858fc966095df72232, fcac3d8c040d49fb8a4aa34a0728264e; no voters:
I20260924 16:12:39.874328 12858 raft_consensus.cc:2781] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:39.874552 12858 raft_consensus.cc:686] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [term 1 LEADER]: Becoming Leader. State: Replica: 269fe5a1dbc04f858fc966095df72232, State: Running, Role: LEADER
I20260924 16:12:39.874895 12858 consensus_queue.cc:237] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcac3d8c040d49fb8a4aa34a0728264e" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35517 } } peers { permanent_uuid: "269fe5a1dbc04f858fc966095df72232" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "272cd765be064903b64769103b8ccfc5" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33315 } }
I20260924 16:12:39.878115 12375 catalog_manager.cc:5495] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 reported cstate change: term changed from 0 to 1, leader changed from <none> to 269fe5a1dbc04f858fc966095df72232 (127.1.253.65). New cstate: current_term: 1 leader_uuid: "269fe5a1dbc04f858fc966095df72232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcac3d8c040d49fb8a4aa34a0728264e" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269fe5a1dbc04f858fc966095df72232" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "272cd765be064903b64769103b8ccfc5" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33315 } health_report { overall_health: UNKNOWN } } }
I20260924 16:12:39.898082 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.894416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.1.253.65:40807)
0924 16:12:39.894572 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:39.894577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:39.894593 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:12:39.894723 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:39.894725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:39.894790 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:39.894908 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:39.895158 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.895673 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.896553 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:39.896774 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:39.896883 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:39.897211 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:39.897664 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:39.897804 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:39.897870 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 16:12:39.897931 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:12:39.904295 12654 raft_consensus.cc:1261] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e [term 1 FOLLOWER]: Refusing update from remote peer 269fe5a1dbc04f858fc966095df72232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:39.904296 12789 raft_consensus.cc:1261] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5 [term 1 FOLLOWER]: Refusing update from remote peer 269fe5a1dbc04f858fc966095df72232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:39.905097 12863 consensus_queue.cc:1035] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcac3d8c040d49fb8a4aa34a0728264e" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 35517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:39.905279 12858 consensus_queue.cc:1035] T 1821287a13b342c987064d519e21346d P 269fe5a1dbc04f858fc966095df72232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "272cd765be064903b64769103b8ccfc5" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 33315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:39.923101 12375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:39.923305 12375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:39.925864 12375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 939d60a6a8884438bf42dcdb411d22eb: Sending DeleteTablet for 3 replicas of tablet 1821287a13b342c987064d519e21346d
I20260924 16:12:39.926674 12634 tablet_service.cc:1514] Processing DeleteTablet for tablet 1821287a13b342c987064d519e21346d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:39 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:36486
I20260924 16:12:39.926752 12769 tablet_service.cc:1514] Processing DeleteTablet for tablet 1821287a13b342c987064d519e21346d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:39 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:49386
I20260924 16:12:39.927076 12500 tablet_service.cc:1514] Processing DeleteTablet for tablet 1821287a13b342c987064d519e21346d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:39 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:33478
I20260924 16:12:39.927502 12876 tablet_replica.cc:330] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e: stopping tablet replica
I20260924 16:12:39.927500 12877 tablet_replica.cc:330] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5: stopping tablet replica
I20260924 16:12:39.927806 12876 raft_consensus.cc:2229] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:39.927806 12877 raft_consensus.cc:2229] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:39.927893 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 12437
I20260924 16:12:39.928109 12877 raft_consensus.cc:2258] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:39.928109 12876 raft_consensus.cc:2258] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:39.930066 12876 ts_tablet_manager.cc:1905] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:39.932335 12876 ts_tablet_manager.cc:1918] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:39.932447 12876 log.cc:1196] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/1821287a13b342c987064d519e21346d
I20260924 16:12:39.932804 12876 ts_tablet_manager.cc:1939] T 1821287a13b342c987064d519e21346d P fcac3d8c040d49fb8a4aa34a0728264e: Deleting consensus metadata
I20260924 16:12:39.933503 12877 ts_tablet_manager.cc:1905] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:39.934084 12359 catalog_manager.cc:4841] TS fcac3d8c040d49fb8a4aa34a0728264e (127.1.253.66:35517): tablet 1821287a13b342c987064d519e21346d (table test-table [id=8a564ed8055646b984a5ffa0ca900251]) successfully deleted
W20260924 16:12:39.935730 12362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:40807 (error 108)
I20260924 16:12:39.935923 12877 ts_tablet_manager.cc:1918] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:39.936043 12877 log.cc:1196] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/1821287a13b342c987064d519e21346d
I20260924 16:12:39.936338 12877 ts_tablet_manager.cc:1939] T 1821287a13b342c987064d519e21346d P 272cd765be064903b64769103b8ccfc5: Deleting consensus metadata
I20260924 16:12:39.936851 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:39.936554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40807 (local address 127.0.0.1:33486)
0924 16:12:39.936690 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:39.936768 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 16:12:39.936853 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 12571
W20260924 16:12:39.937834 12362 catalog_manager.cc:4584] TS 269fe5a1dbc04f858fc966095df72232 (127.1.253.65:40807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1821287a13b342c987064d519e21346d: Network error: Client connection negotiation failed: client connection to 127.1.253.65:40807: connect: Connection refused (error 111)
I20260924 16:12:39.938072 12362 catalog_manager.cc:4841] TS 272cd765be064903b64769103b8ccfc5 (127.1.253.67:33315): tablet 1821287a13b342c987064d519e21346d (table test-table [id=8a564ed8055646b984a5ffa0ca900251]) successfully deleted
I20260924 16:12:39.943651 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 12705
I20260924 16:12:39.949873 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 12343
2026-09-24T16:12:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2325 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:40 dist-test-slave-6wvm krb5kdc[12882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:40 dist-test-slave-6wvm krb5kdc[12882](info): set up 2 sockets
Sep 24 16:12:40 dist-test-slave-6wvm krb5kdc[12882](info): commencing operation
krb5kdc: starting...
W20260924 16:12:41.003113 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:12:41 dist-test-slave-6wvm krb5kdc[12882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.253.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.253.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/rangeradmin_127.1.253.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/rangeradmin_127.1.253.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.253.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.253.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/rangerlookup_127.1.253.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/rangerlookup_127.1.253.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.253.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.253.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/HTTP_127.1.253.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/HTTP_127.1.253.84@KRBTEST.COM.keytab.
I20260924 16:12:41.175359 2037 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6Pu02A/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/postgres -l logfile start
2026-09-24 16:12:43.626 UTC [12912] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 16:12:43.627 UTC [12912] LOG: listening on IPv4 address "127.1.253.84", port 54515
2026-09-24 16:12:43.630 UTC [12912] LOG: listening on Unix socket "/tmp/.s.PGSQL.54515"
2026-09-24 16:12:43.651 UTC [12915] LOG: database system was shut down at 2026-09-24 16:12:42 UTC
2026-09-24 16:12:43.654 UTC [12912] LOG: database system is ready to accept connections
I20260924 16:12:44.610989 2037 mini_postgres.cc:96] Postgres bound to 54515
2026-09-24 16:12:44.618 UTC [12923] FATAL: database "slave" does not exist
127.1.253.84:54515 - accepting connections
I20260924 16:12:44.619053 2037 mini_ranger.cc:161] Starting Ranger...
I20260924 16:12:44.634665 2037 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 16:12:45.093752 2037 mini_ranger.cc:87] Created ranger Postgres database
I20260924 16:12:45.093873 2037 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-admin
2026-09-24 16:12:45,399 [I] DB FLAVOR :POSTGRES
2026-09-24 16:12:45,399 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:12:45,399 [I] Checking connection..
2026-09-24 16:12:45,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 16:12:45,651 [I] Checking connection passed.
2026-09-24 16:12:45,651 [I] --------- Verifying version history table ---------
2026-09-24 16:12:45,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:12:45,912 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:12:45,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:12:46,171 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:12:46,171 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 16:12:46,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 16:12:46.414 UTC [12987] WARNING: there is no transaction in progress
2026-09-24 16:12:46,421 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 16:12:46,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:12:46,670 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 16:12:46,670 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:12:46,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 16:12:46,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 16:12:47,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6wvm.c.gcp-upstream.internal','N') ;"
2026-09-24 16:12:47,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:12:47,626 [I] Table x_portal_user does not exist in database ranger
2026-09-24 16:12:47,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:12:47,870 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 16:12:47,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:12:48,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:12:48,340 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 16:12:48,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 16:12:49.286 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.302 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.319 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.334 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.349 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.370 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.425 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.432 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.447 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.457 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.467 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.481 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.491 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.498 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.504 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:49.510 UTC [13095] WARNING: there is no transaction in progress
2026-09-24 16:12:50,148 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 16:12:50,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 16:12:50,391 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 16:12:50,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:12:50,638 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 16:12:50,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:12:50,901 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 16:12:50,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:12:51,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:12:51,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6wvm.c.gcp-upstream.internal';"
2026-09-24 16:12:51,666 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 16:12:51,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 16:12:51,898 [I] DB_PATCHES have already been applied
I20260924 16:12:51.904307 2037 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-admin:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6Pu02A/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 16:12:51.904379 2037 mini_ranger.cc:193] Using host: 127.1.253.84
I20260924 16:12:51.909274 2037 mini_ranger.cc:227] Ranger admin URL: http://127.1.253.84:35197
Sep 24, 2026 4:12:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:12:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:12:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:12:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:12:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:12:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:12:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 16:12:53.052978 2037 mini_ranger.cc:160] Time spent starting Ranger: real 8.434s user 0.000s sys 0.011s
Sep 24, 2026 4:13:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:13:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 4:13:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:13:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:13:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:13:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 16:13:15.657984 2037 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.253.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.253.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/rangerkms_127.1.253.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/rangerkms_127.1.253.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.253.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.253.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.253.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/HTTP_127.1.253.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.253.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/HTTP_127.1.253.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 16:13:15 dist-test-slave-6wvm krb5kdc[12882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266395, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 16:13:15.750295 2037 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 16:13:15.760071 2037 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 16:13:16.689518 2037 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 16:13:16.689620 2037 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-kms
I20260924 16:13:16.689733 2037 mini_ranger_kms.cc:227] Using postgres at 127.1.253.84:54515
2026-09-24 16:13:17,784 [I] DB FLAVOR :POSTGRES
2026-09-24 16:13:17,785 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:13:17,785 [I] Checking connection
2026-09-24 16:13:17,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 16:13:18,100 [I] connection success
2026-09-24 16:13:18,100 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 16:13:18,101 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 16:13:18,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 16:13:18,377 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 16:13:18,377 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:13:18,377 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 16:13:18,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:54515/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 16:13:18,643 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 16:13:19.146095 2037 mini_ranger_kms.cc:310] Created kms service
I20260924 16:13:19.391348 2037 mini_ranger_kms.cc:326] Created kudu user
I20260924 16:13:19.436930 2037 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 16:13:19 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266395, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:13:19.879980 2037 mini_ranger_kms.cc:384] Added ranger policy
I20260924 16:13:19.880169 2037 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-kms:/tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 16:13:19.880198 2037 mini_ranger_kms.cc:240] Using host: 127.1.253.84
I20260924 16:13:19.884621 2037 mini_ranger_kms.cc:276] Ranger KMS PID: 13276
I20260924 16:13:19.884792 2037 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.253.84:58209
Sep 24, 2026 4:13:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:13:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:13:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:13:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:13:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:13:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 16:13:22.112910 2037 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.363s user 0.005s sys 0.403s
I20260924 16:13:22.113039 2037 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 16:13:22.113200 2037 mini_ranger_kms.cc:401] 127.1.253.84:58209/kms/v1/keys
Sep 24 16:13:22 dist-test-slave-6wvm krb5kdc[12882](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266402, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.253.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 16:13:22 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266402, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.253.84@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
Sep 24 16:13:23 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266402, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.253.84@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
Sep 24 16:13:25 dist-test-slave-6wvm krb5kdc[12882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:13:25.611222 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:38949
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:38949
--ranger_config_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.253.84:58209/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:25.749452 13326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:25.749776 13326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:25.749818 13326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:25.754065 13326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:13:25.754158 13326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:25.754179 13326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:25.754195 13326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:13:25.754211 13326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:13:25.760717 13326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.253.84:58209/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:38949
--ranger_config_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:38949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:13:25.761991 13326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:25.763175 13326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:25.773835 13332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:25.773854 13334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:25.774289 13326 server_base.cc:1034] running on GCE node
W20260924 16:13:25.773965 13331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:25.775061 13326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:25.776305 13326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:25.777493 13326 hybrid_clock.cc:648] HybridClock initialized: now 1790266405777464 us; error 45 us; skew 500 ppm
Sep 24 16:13:25 dist-test-slave-6wvm krb5kdc[12882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266405, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:25.784876 13326 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:13:25.785897 13326 webserver.cc:460] Webserver started at http://127.1.253.126:45725/ using document root <none> and password file <none>
I20260924 16:13:25.786530 13326 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:25.786586 13326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:25.786804 13326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:13:25 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266405, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:13:25.850584 13326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "e805c25782c94381a79db0dc5031c4c7"
format_stamp: "Formatted at 2026-09-24 16:13:25 on dist-test-slave-6wvm"
server_key: "50cf0b8dd9c0c3ec84c62086581ac2df"
server_key_iv: "ffdbd70581ff6835ec9fce3f2aa3a8c0"
server_key_version: "kuduclusterkey@0"
I20260924 16:13:25.851267 13326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "e805c25782c94381a79db0dc5031c4c7"
format_stamp: "Formatted at 2026-09-24 16:13:25 on dist-test-slave-6wvm"
server_key: "50cf0b8dd9c0c3ec84c62086581ac2df"
server_key_iv: "ffdbd70581ff6835ec9fce3f2aa3a8c0"
server_key_version: "kuduclusterkey@0"
I20260924 16:13:25.855545 13326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 16:13:25.874002 13341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:25.875805 13326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:13:25.875958 13326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "e805c25782c94381a79db0dc5031c4c7"
format_stamp: "Formatted at 2026-09-24 16:13:25 on dist-test-slave-6wvm"
server_key: "50cf0b8dd9c0c3ec84c62086581ac2df"
server_key_iv: "ffdbd70581ff6835ec9fce3f2aa3a8c0"
server_key_version: "kuduclusterkey@0"
I20260924 16:13:25.876097 13326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:25.907780 13326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:25.911441 13326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:25.911667 13326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:25.920300 13326 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:38949
I20260924 16:13:25.920327 13393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:38949 every 8 connection(s)
I20260924 16:13:25.921521 13326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:13:25.925275 13394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:25.931105 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 13326
I20260924 16:13:25.931253 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:13:25.931556 2037 external_mini_cluster.cc:1428] Setting key 7ae521a7f3eae9c6aeec0aac7230e8f5
I20260924 16:13:25.932900 13394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7: Bootstrap starting.
I20260924 16:13:25.935819 13394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:25.936750 13394 log.cc:824] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:25.939178 13394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7: No bootstrap required, opened a new log
Sep 24 16:13:25 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:13:25.943881 13394 raft_consensus.cc:348] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e805c25782c94381a79db0dc5031c4c7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38949 } }
I20260924 16:13:25.944204 13394 raft_consensus.cc:374] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:25.944262 13394 raft_consensus.cc:729] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e805c25782c94381a79db0dc5031c4c7, State: Initialized, Role: FOLLOWER
I20260924 16:13:25.944790 13394 consensus_queue.cc:260] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e805c25782c94381a79db0dc5031c4c7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38949 } }
I20260924 16:13:25.944954 13394 raft_consensus.cc:388] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:13:25.945019 13394 raft_consensus.cc:482] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:13:25.945111 13394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:25.946303 13394 raft_consensus.cc:504] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e805c25782c94381a79db0dc5031c4c7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38949 } }
I20260924 16:13:25.946705 13394 leader_election.cc:302] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e805c25782c94381a79db0dc5031c4c7; no voters:
I20260924 16:13:25.947065 13394 leader_election.cc:288] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:13:25.947983 13397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:25.933174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40706 (local address 127.1.253.126:38949)
0924 16:13:25.933611 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:25.933627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:25.933663 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:13:25.934414 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:25.934433 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:25.934781 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:25.935348 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:25.936416 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.937761 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.938771 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.939316 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.939521 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:25.942327 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:25.942335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:25.942351 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:25.942384 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:25.944475 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:25.945113 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:25.945121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:25.945128 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:25.945288 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:25.945643 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:25.945649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:25.945651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:25.946113 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:25.946382 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:25.946770 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 16:13:25.947117 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":147,"threads_started":1}
I20260924 16:13:25.948400 13394 sys_catalog.cc:564] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:13:25.950393 13400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:25.950706 13400 raft_consensus.cc:686] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e805c25782c94381a79db0dc5031c4c7, State: Running, Role: LEADER
I20260924 16:13:25.951069 13400 consensus_queue.cc:237] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e805c25782c94381a79db0dc5031c4c7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38949 } }
I20260924 16:13:25.952276 13400 sys_catalog.cc:455] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e805c25782c94381a79db0dc5031c4c7. Latest consensus state: current_term: 1 leader_uuid: "e805c25782c94381a79db0dc5031c4c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e805c25782c94381a79db0dc5031c4c7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38949 } } }
I20260924 16:13:25.952430 13400 sys_catalog.cc:458] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:25.953081 13394 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:13:25.954485 13411 sys_catalog.cc:455] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e805c25782c94381a79db0dc5031c4c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e805c25782c94381a79db0dc5031c4c7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38949 } } }
I20260924 16:13:25.954602 13411 sys_catalog.cc:458] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:25.958776 13413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:13:25.963291 13413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:13:25.971504 13413 catalog_manager.cc:1347] Generated new cluster ID: 5503d2f0b3ff4ea89563050b086c6bda
I20260924 16:13:25.971594 13413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:13:25.983326 13413 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:13:25.984920 13413 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:13:25.994553 13413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7: Generated new TSK 0
I20260924 16:13:25.995590 13413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:13:26.922032 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.253.84:58209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38949
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:27.085289 13439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:27.085618 13439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:27.085709 13439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:27.090013 13439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:27.090114 13439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:27.090214 13439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:13:27.096310 13439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.253.84:58209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:13:27.097666 13439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:27.098862 13439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:27.109709 13445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:27.109731 13444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:27.109822 13447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:27.110502 13439 server_base.cc:1034] running on GCE node
I20260924 16:13:27.111006 13439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:27.111730 13439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:27.112984 13439 hybrid_clock.cc:648] HybridClock initialized: now 1790266407112957 us; error 34 us; skew 500 ppm
Sep 24 16:13:27 dist-test-slave-6wvm krb5kdc[12882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:27.116765 13439 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:13:27.118021 13439 webserver.cc:460] Webserver started at http://127.1.253.65:35835/ using document root <none> and password file <none>
I20260924 16:13:27.118824 13439 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:27.118921 13439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:27.119191 13439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:13:27 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:13:27.137315 13439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "3f499bc7c479446893f72f8009ef089e"
format_stamp: "Formatted at 2026-09-24 16:13:27 on dist-test-slave-6wvm"
server_key: "bd357f662c1b39ad2940fc134982b7a1"
server_key_iv: "253ee5b6e4db5ecd1348fc16c6fad095"
server_key_version: "kuduclusterkey@0"
I20260924 16:13:27.138113 13439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "3f499bc7c479446893f72f8009ef089e"
format_stamp: "Formatted at 2026-09-24 16:13:27 on dist-test-slave-6wvm"
server_key: "bd357f662c1b39ad2940fc134982b7a1"
server_key_iv: "253ee5b6e4db5ecd1348fc16c6fad095"
server_key_version: "kuduclusterkey@0"
I20260924 16:13:27.145269 13439 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260924 16:13:27.170770 13456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:27.172797 13439 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.000s
I20260924 16:13:27.172981 13439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "3f499bc7c479446893f72f8009ef089e"
format_stamp: "Formatted at 2026-09-24 16:13:27 on dist-test-slave-6wvm"
server_key: "bd357f662c1b39ad2940fc134982b7a1"
server_key_iv: "253ee5b6e4db5ecd1348fc16c6fad095"
server_key_version: "kuduclusterkey@0"
I20260924 16:13:27.173149 13439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:27.210000 13439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:27.214285 13439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:27.214486 13439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:27.215212 13439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:27.216357 13439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:27.216408 13439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:27.216461 13439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:27.216485 13439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:27.235545 13439 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:38993
I20260924 16:13:27.237001 13439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:13:27.242844 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 13439
I20260924 16:13:27.242980 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:13:27.243328 2037 external_mini_cluster.cc:1428] Setting key 971f554c06311387036ad63963a89d8b
I20260924 16:13:27.267403 13570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:38993 every 8 connection(s)
Sep 24 16:13:27 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:13:27.296870 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:27.265300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:40973 (local address 127.1.253.126:38949)
0924 16:13:27.266401 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:27.266406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:27.269786 (+ 3380us) server_negotiation.cc:407] Connection header received
0924 16:13:27.271532 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:27.271539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:27.271623 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:27.271698 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:27.273859 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.274826 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.276691 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.276930 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.277063 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:27.282021 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:27.282026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:27.282029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:27.282068 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:27.284205 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:27.286412 (+ 2207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:27.286417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:27.286419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:27.286559 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:27.294317 (+ 7758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:27.294321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:27.294323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:27.294649 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:27.294759 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:27.296442 (+ 1683us) server_negotiation.cc:299] Negotiation successful
0924 16:13:27.296622 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":104,"threads_started":1}
I20260924 16:13:27.299132 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:27.265845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38949 (local address 127.1.253.65:40973)
0924 16:13:27.269565 (+ 3720us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:27.269638 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:27.271206 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:27.272267 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:27.272281 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:27.272795 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:27.273578 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:27.273602 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.275021 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.275028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:27.276544 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:27.276555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.277114 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.277120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:27.277343 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:27.278105 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:27.278130 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:27.281784 (+ 3654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:27.284417 (+ 2633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:27.284425 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:27.284447 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:27.284836 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:27.293767 (+ 8931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:27.293770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:27.293773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:27.294081 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:27.294866 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:27.294874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:27.295484 (+ 610us) client_negotiation.cc:769] Sending connection context
0924 16:13:27.297116 (+ 1632us) client_negotiation.cc:240] Negotiation successful
0924 16:13:27.297668 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3414,"thread_start_us":281,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
I20260924 16:13:27.301112 13571 heartbeater.cc:347] Connected to a master server at 127.1.253.126:38949
I20260924 16:13:27.301421 13571 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:27.302131 13571 heartbeater.cc:511] Master 127.1.253.126:38949 requested a full tablet report, sending...
I20260924 16:13:27.304236 13358 ts_manager.cc:194] Registered new tserver with Master: 3f499bc7c479446893f72f8009ef089e (127.1.253.65:38993)
I20260924 16:13:27.306070 13358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:40973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:13:27.321734 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.253.84:58209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38949
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:13:27 dist-test-slave-6wvm krb5kdc[12882](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 16:13:27.541119 13579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:27.541501 13579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:27.541572 13579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:27.547713 13579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:27.547801 13579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:27.547886 13579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:13:27.555711 13579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.253.84:58209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:13:27.557211 13579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:27.558640 13579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:27.573489 13586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:27.573508 13585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:27.573863 13588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:27.575052 13579 server_base.cc:1034] running on GCE node
I20260924 16:13:27.575620 13579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:27.576493 13579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:27.577723 13579 hybrid_clock.cc:648] HybridClock initialized: now 1790266407577694 us; error 34 us; skew 500 ppm
Sep 24 16:13:27 dist-test-slave-6wvm krb5kdc[12882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:27.582955 13579 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:13:27.584724 13579 webserver.cc:460] Webserver started at http://127.1.253.66:35689/ using document root <none> and password file <none>
I20260924 16:13:27.585759 13579 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:27.585960 13579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:27.586409 13579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:13:27 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:13:27.606487 13579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b"
format_stamp: "Formatted at 2026-09-24 16:13:27 on dist-test-slave-6wvm"
server_key: "a86c22c20c1e21c7cdb757dd59971227"
server_key_iv: "d62722673917f283b684148b1b7e03b6"
server_key_version: "kuduclusterkey@0"
I20260924 16:13:27.607883 13579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b"
format_stamp: "Formatted at 2026-09-24 16:13:27 on dist-test-slave-6wvm"
server_key: "a86c22c20c1e21c7cdb757dd59971227"
server_key_iv: "d62722673917f283b684148b1b7e03b6"
server_key_version: "kuduclusterkey@0"
I20260924 16:13:27.612703 13579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:13:27.630674 13595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:27.632625 13579 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260924 16:13:27.632792 13579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b"
format_stamp: "Formatted at 2026-09-24 16:13:27 on dist-test-slave-6wvm"
server_key: "a86c22c20c1e21c7cdb757dd59971227"
server_key_iv: "d62722673917f283b684148b1b7e03b6"
server_key_version: "kuduclusterkey@0"
I20260924 16:13:27.632977 13579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:27.657351 13579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:27.661433 13579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:27.661643 13579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:27.662590 13579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:27.664018 13579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:27.664088 13579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:27.664155 13579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:27.664188 13579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:27.686920 13579 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:41429
I20260924 16:13:27.688271 13579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:13:27.693277 13708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:41429 every 8 connection(s)
I20260924 16:13:27.698753 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 13579
I20260924 16:13:27.698887 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:13:27.699247 2037 external_mini_cluster.cc:1428] Setting key 824608e826340bede79d7df773bd380d
Sep 24 16:13:27 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:13:27.709918 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:27.691809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38949 (local address 127.1.253.66:58967)
0924 16:13:27.693701 (+ 1892us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:27.693763 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:27.694882 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:27.695551 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:27.695560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:27.695923 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:27.696615 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:27.696636 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.697772 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.697776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:27.699238 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:27.699250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.700041 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.700046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:27.700259 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:27.701013 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:27.701036 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:27.702857 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:27.705301 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:27.705308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:27.705331 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:27.705698 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:27.706294 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:27.706298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:27.706300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:27.706500 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:27.707189 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:27.707196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:27.707641 (+ 445us) client_negotiation.cc:769] Sending connection context
0924 16:13:27.707918 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 16:13:27.708434 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1647,"thread_start_us":132,"threads_started":1}
I20260924 16:13:27.710762 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:27.692999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:58967 (local address 127.1.253.126:38949)
0924 16:13:27.693133 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:27.693138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:27.693903 (+ 765us) server_negotiation.cc:407] Connection header received
0924 16:13:27.695153 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:27.695158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:27.695223 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:27.695338 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:27.696805 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.697613 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.699382 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.699738 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.699823 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:27.703062 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:27.703066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:27.703069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:27.703098 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:27.705155 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:27.705825 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:27.705829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:27.705831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:27.705922 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:27.706637 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:27.706641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:27.706642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:27.706976 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:27.707098 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:27.710417 (+ 3319us) server_negotiation.cc:299] Negotiation successful
0924 16:13:27.710585 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:13:27.712776 13709 heartbeater.cc:347] Connected to a master server at 127.1.253.126:38949
I20260924 16:13:27.713073 13709 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:27.713738 13709 heartbeater.cc:511] Master 127.1.253.126:38949 requested a full tablet report, sending...
I20260924 16:13:27.715005 13358 ts_manager.cc:194] Registered new tserver with Master: a0e4042f6f1548ef803a9f0ad3e20a6b (127.1.253.66:41429)
I20260924 16:13:27.715804 13358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:58967
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:13:27.750974 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.253.84:58209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38949
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:27.958123 13714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:27.958716 13714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:27.958863 13714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:27.965031 13714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:27.965186 13714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:27.965409 13714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:13:27.973686 13714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.253.84:58209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:13:27.975426 13714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:27.976840 13714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:27.991798 13720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:27.992413 13719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:27.993302 13722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:27.994222 13714 server_base.cc:1034] running on GCE node
I20260924 16:13:27.994968 13714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:27.995848 13714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:27.997215 13714 hybrid_clock.cc:648] HybridClock initialized: now 1790266407997144 us; error 88 us; skew 500 ppm
Sep 24 16:13:27 dist-test-slave-6wvm krb5kdc[12882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:28.002106 13714 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:13:28.003455 13714 webserver.cc:460] Webserver started at http://127.1.253.67:35455/ using document root <none> and password file <none>
I20260924 16:13:28.004305 13714 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:28.004458 13714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:28.004750 13714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:13:28 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:13:28.028673 13714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "4e86c301c70c4f4f9d21b0bb2352361b"
format_stamp: "Formatted at 2026-09-24 16:13:28 on dist-test-slave-6wvm"
server_key: "c80b618ca5b5296b2b5c92c0568bee09"
server_key_iv: "3880ef36b3ca7edfde8c3fb38cd05173"
server_key_version: "kuduclusterkey@0"
I20260924 16:13:28.037384 13714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "4e86c301c70c4f4f9d21b0bb2352361b"
format_stamp: "Formatted at 2026-09-24 16:13:28 on dist-test-slave-6wvm"
server_key: "c80b618ca5b5296b2b5c92c0568bee09"
server_key_iv: "3880ef36b3ca7edfde8c3fb38cd05173"
server_key_version: "kuduclusterkey@0"
I20260924 16:13:28.043792 13714 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
Sep 24 16:13:28 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:13:28.071393 13729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:28.073649 13714 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:13:28.073818 13714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "4e86c301c70c4f4f9d21b0bb2352361b"
format_stamp: "Formatted at 2026-09-24 16:13:28 on dist-test-slave-6wvm"
server_key: "c80b618ca5b5296b2b5c92c0568bee09"
server_key_iv: "3880ef36b3ca7edfde8c3fb38cd05173"
server_key_version: "kuduclusterkey@0"
I20260924 16:13:28.073978 13714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:28.117655 13714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:28.122074 13714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:28.122383 13714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:28.123250 13714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:28.125316 13714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:28.125428 13714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 16:13:28.125536 13714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:28.125598 13714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:28.140924 13714 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:44107
I20260924 16:13:28.142261 13714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:13:28.144214 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 13714
I20260924 16:13:28.144341 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:13:28.144672 2037 external_mini_cluster.cc:1428] Setting key e2214ba68f9f03410176b8ea7ca1c423
I20260924 16:13:28.144680 13842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:44107 every 8 connection(s)
Sep 24 16:13:28 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:13:28.163753 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.148296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:36467 (local address 127.1.253.126:38949)
0924 16:13:28.148431 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:28.148435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:28.149285 (+ 850us) server_negotiation.cc:407] Connection header received
0924 16:13:28.150650 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:28.150654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:28.150710 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:28.150761 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:28.152863 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.153695 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.155542 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.155900 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.155976 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.159281 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:28.159285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:28.159288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:28.159318 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:28.161177 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:28.161839 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:28.161843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:28.161845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:28.161936 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:28.162364 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:28.162368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:28.162370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:28.162603 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:28.162698 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:28.163428 (+ 730us) server_negotiation.cc:299] Negotiation successful
0924 16:13:28.163569 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:13:28.165745 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.147178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38949 (local address 127.1.253.67:36467)
0924 16:13:28.149123 (+ 1945us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:28.149182 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:28.150394 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:28.151197 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:28.151207 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:28.151808 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:28.152599 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.152621 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.153877 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.153880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.155395 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.155407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.156198 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.156202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.156411 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.157212 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:28.157234 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:28.159076 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:28.161329 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:28.161336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:28.161356 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:28.161720 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:28.162040 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:28.162043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:28.162045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:28.162228 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:28.162770 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:28.162776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:28.163135 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 16:13:28.164222 (+ 1087us) client_negotiation.cc:240] Negotiation successful
0924 16:13:28.164604 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1702,"thread_start_us":153,"threads_started":1}
I20260924 16:13:28.167438 13843 heartbeater.cc:347] Connected to a master server at 127.1.253.126:38949
I20260924 16:13:28.167714 13843 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:28.168329 13843 heartbeater.cc:511] Master 127.1.253.126:38949 requested a full tablet report, sending...
I20260924 16:13:28.169605 13358 ts_manager.cc:194] Registered new tserver with Master: 4e86c301c70c4f4f9d21b0bb2352361b (127.1.253.67:44107)
I20260924 16:13:28.170385 13358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:36467
I20260924 16:13:28.171555 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:13:28.186734 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.176137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33452 (local address 127.1.253.126:38949)
0924 16:13:28.176258 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:28.176263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:28.176910 (+ 647us) server_negotiation.cc:407] Connection header received
0924 16:13:28.177095 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:28.177099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:28.177148 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:28.177200 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:28.178182 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.179018 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.180329 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.180774 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.180869 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.183049 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:28.183053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:28.183056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:28.183083 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:28.184813 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:28.185325 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:28.185329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:28.185331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:28.185413 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:28.185721 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:28.185724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:28.185726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:28.185935 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:28.186038 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:28.186424 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 16:13:28.186550 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:13:28.190747 13358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:13:28.203964 13358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:13:28.268597 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.244302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:38993 (local address 127.0.0.1:34978)
0924 16:13:28.256592 (+ 12290us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:28.256606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:28.256711 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:28.263137 (+ 6426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:28.263140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:28.263158 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:28.263268 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.263274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.264516 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.264519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.265748 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.265759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.267606 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.267619 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.267836 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.267944 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 16:13:28.268159 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 16:13:28.268403 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10339,"thread_start_us":136,"threads_started":1}
I20260924 16:13:28.269767 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.244435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34978 (local address 127.1.253.65:38993)
0924 16:13:28.262483 (+ 18048us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:28.262490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:28.262514 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:13:28.262587 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:28.262592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:28.262824 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:28.262989 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:28.263395 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.264391 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.265886 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.267071 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.267234 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.267499 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:28.269513 (+ 2014us) server_negotiation.cc:299] Negotiation successful
0924 16:13:28.269631 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17917,"thread_start_us":75,"threads_started":1}
I20260924 16:13:28.273335 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.246830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:41429 (local address 127.0.0.1:59744)
0924 16:13:28.255863 (+ 9033us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:28.255935 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:28.256080 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:28.259592 (+ 3512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:28.259595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:28.259614 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:28.259765 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.259772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.262330 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.262334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.270390 (+ 8056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.270400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.271993 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.271996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.272884 (+ 888us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.272977 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 16:13:28.273084 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:13:28.273197 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8800,"thread_start_us":126,"threads_started":1}
I20260924 16:13:28.273403 13502 tablet_service.cc:1467] Processing CreateTablet for tablet 04992c82ec6345fd801d825cd985d7a3 (DEFAULT_TABLE table=test-table [id=e316bdba19864591a4913301810fffba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:28.274726 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.248225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:44107 (local address 127.0.0.1:56040)
0924 16:13:28.257618 (+ 9393us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:28.257632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:28.257737 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:28.258531 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:28.258537 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:28.258560 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:28.259991 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.259998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.265110 (+ 5112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.265113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.270502 (+ 5389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.270506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.274293 (+ 3787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.274297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.274411 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.274438 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:13:28.274511 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 16:13:28.274582 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9332,"thread_start_us":150,"threads_started":1}
I20260924 16:13:28.274761 13502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04992c82ec6345fd801d825cd985d7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:28.275561 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.246472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.1.253.66:41429)
0924 16:13:28.254506 (+ 8034us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:28.254515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:28.256818 (+ 2303us) server_negotiation.cc:407] Connection header received
0924 16:13:28.256913 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:28.256919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:28.257165 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:28.257321 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:28.260123 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.261404 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.270621 (+ 9217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.271452 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.271565 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.271740 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:28.275300 (+ 3560us) server_negotiation.cc:299] Negotiation successful
0924 16:13:28.275421 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7891,"thread_start_us":128,"threads_started":1}
I20260924 16:13:28.280133 13633 tablet_service.cc:1467] Processing CreateTablet for tablet 04992c82ec6345fd801d825cd985d7a3 (DEFAULT_TABLE table=test-table [id=e316bdba19864591a4913301810fffba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:28.281699 13633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04992c82ec6345fd801d825cd985d7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:28.290693 13862 tablet_bootstrap.cc:492] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e: Bootstrap starting.
I20260924 16:13:28.292224 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.248748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.1.253.67:44107)
0924 16:13:28.256500 (+ 7752us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:28.256535 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:28.257832 (+ 1297us) server_negotiation.cc:407] Connection header received
0924 16:13:28.257915 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:28.257920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:28.258117 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:28.258360 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:28.261555 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.264980 (+ 3425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.271880 (+ 6900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.273929 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.274050 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.274195 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:28.275838 (+ 1643us) server_negotiation.cc:299] Negotiation successful
0924 16:13:28.275951 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7632,"thread_start_us":213,"threads_started":1}
I20260924 16:13:28.293720 13862 tablet_bootstrap.cc:654] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:28.295185 13862 log.cc:824] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:28.296393 13765 tablet_service.cc:1467] Processing CreateTablet for tablet 04992c82ec6345fd801d825cd985d7a3 (DEFAULT_TABLE table=test-table [id=e316bdba19864591a4913301810fffba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:28.298337 13862 tablet_bootstrap.cc:492] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e: No bootstrap required, opened a new log
I20260924 16:13:28.298568 13862 ts_tablet_manager.cc:1397] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260924 16:13:28.299285 13765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04992c82ec6345fd801d825cd985d7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:28.300338 13863 tablet_bootstrap.cc:492] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b: Bootstrap starting.
I20260924 16:13:28.302949 13863 tablet_bootstrap.cc:654] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:28.303471 13862 raft_consensus.cc:348] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f499bc7c479446893f72f8009ef089e" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38993 } } peers { permanent_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41429 } } peers { permanent_uuid: "4e86c301c70c4f4f9d21b0bb2352361b" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44107 } }
I20260924 16:13:28.303772 13862 raft_consensus.cc:374] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:28.303826 13862 raft_consensus.cc:729] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f499bc7c479446893f72f8009ef089e, State: Initialized, Role: FOLLOWER
I20260924 16:13:28.304005 13863 log.cc:824] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:28.304456 13862 consensus_queue.cc:260] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f499bc7c479446893f72f8009ef089e" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38993 } } peers { permanent_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41429 } } peers { permanent_uuid: "4e86c301c70c4f4f9d21b0bb2352361b" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44107 } }
I20260924 16:13:28.305766 13862 ts_tablet_manager.cc:1428] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260924 16:13:28.306284 13571 heartbeater.cc:503] Master 127.1.253.126:38949 was elected leader, sending a full tablet report...
I20260924 16:13:28.309926 13867 tablet_bootstrap.cc:492] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b: Bootstrap starting.
I20260924 16:13:28.312245 13863 tablet_bootstrap.cc:492] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b: No bootstrap required, opened a new log
I20260924 16:13:28.312559 13863 ts_tablet_manager.cc:1397] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260924 16:13:28.314109 13867 tablet_bootstrap.cc:654] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:28.315080 13867 log.cc:824] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:28.317293 13867 tablet_bootstrap.cc:492] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b: No bootstrap required, opened a new log
I20260924 16:13:28.317494 13867 ts_tablet_manager.cc:1397] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260924 16:13:28.317384 13863 raft_consensus.cc:348] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f499bc7c479446893f72f8009ef089e" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38993 } } peers { permanent_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41429 } } peers { permanent_uuid: "4e86c301c70c4f4f9d21b0bb2352361b" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44107 } }
I20260924 16:13:28.317674 13863 raft_consensus.cc:374] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:28.317731 13863 raft_consensus.cc:729] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e4042f6f1548ef803a9f0ad3e20a6b, State: Initialized, Role: FOLLOWER
I20260924 16:13:28.318223 13863 consensus_queue.cc:260] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f499bc7c479446893f72f8009ef089e" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38993 } } peers { permanent_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41429 } } peers { permanent_uuid: "4e86c301c70c4f4f9d21b0bb2352361b" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44107 } }
I20260924 16:13:28.319397 13863 ts_tablet_manager.cc:1428] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260924 16:13:28.319841 13709 heartbeater.cc:503] Master 127.1.253.126:38949 was elected leader, sending a full tablet report...
I20260924 16:13:28.323851 13867 raft_consensus.cc:348] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f499bc7c479446893f72f8009ef089e" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38993 } } peers { permanent_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41429 } } peers { permanent_uuid: "4e86c301c70c4f4f9d21b0bb2352361b" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44107 } }
I20260924 16:13:28.324277 13867 raft_consensus.cc:374] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:28.324373 13867 raft_consensus.cc:729] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e86c301c70c4f4f9d21b0bb2352361b, State: Initialized, Role: FOLLOWER
I20260924 16:13:28.324872 13867 consensus_queue.cc:260] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f499bc7c479446893f72f8009ef089e" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38993 } } peers { permanent_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41429 } } peers { permanent_uuid: "4e86c301c70c4f4f9d21b0bb2352361b" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44107 } }
I20260924 16:13:28.325593 13843 heartbeater.cc:503] Master 127.1.253.126:38949 was elected leader, sending a full tablet report...
I20260924 16:13:28.328670 13867 ts_tablet_manager.cc:1428] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
W20260924 16:13:28.399221 13844 tablet.cc:2369] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:13:28.418968 13866 raft_consensus.cc:482] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:13:28.419323 13866 raft_consensus.cc:504] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f499bc7c479446893f72f8009ef089e" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38993 } } peers { permanent_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41429 } } peers { permanent_uuid: "4e86c301c70c4f4f9d21b0bb2352361b" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44107 } }
I20260924 16:13:28.420689 13866 leader_election.cc:288] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e4042f6f1548ef803a9f0ad3e20a6b (127.1.253.66:41429), 4e86c301c70c4f4f9d21b0bb2352361b (127.1.253.67:44107)
I20260924 16:13:28.428184 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.421170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:55543 (local address 127.1.253.66:41429)
0924 16:13:28.421334 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:28.421341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:28.422031 (+ 690us) server_negotiation.cc:407] Connection header received
0924 16:13:28.422324 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:28.422330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:28.422395 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:28.422444 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:28.423474 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.424229 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.426324 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.427011 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.427096 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.427152 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:28.427933 (+ 781us) server_negotiation.cc:299] Negotiation successful
0924 16:13:28.428040 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:13:28.429095 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.420959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:41429 (local address 127.1.253.65:55543)
0924 16:13:28.421896 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:28.421912 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:28.422137 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:28.422605 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:28.422607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:28.422627 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:28.423332 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.423344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.424391 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.424394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.426151 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.426160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.427266 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.427269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.427785 (+ 516us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.427811 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 16:13:28.428663 (+ 852us) client_negotiation.cc:240] Negotiation successful
0924 16:13:28.428763 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":172,"thread_start_us":157,"threads_started":1}
I20260924 16:13:28.429546 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.421798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:48029 (local address 127.1.253.67:44107)
0924 16:13:28.421932 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:28.421937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:28.422192 (+ 255us) server_negotiation.cc:407] Connection header received
0924 16:13:28.422413 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:28.422419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:28.422469 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:28.422513 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:28.423163 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.424001 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.426324 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.426991 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.427096 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.427152 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:28.429279 (+ 2127us) server_negotiation.cc:299] Negotiation successful
0924 16:13:28.429373 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:13:28.430034 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.421476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:44107 (local address 127.1.253.65:48029)
0924 16:13:28.422084 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:28.422097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:28.422296 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:28.422636 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:28.422638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:28.422652 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:28.423015 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.423025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.424263 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.424268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.426148 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.426160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.427266 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.427269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.427523 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.427553 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:13:28.429741 (+ 2188us) client_negotiation.cc:240] Negotiation successful
0924 16:13:28.429828 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":240,"spinlock_wait_cycles":10624,"thread_start_us":99,"threads_started":1}
I20260924 16:13:28.430076 13656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04992c82ec6345fd801d825cd985d7a3" candidate_uuid: "3f499bc7c479446893f72f8009ef089e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b" is_pre_election: true
I20260924 16:13:28.430510 13656 raft_consensus.cc:2454] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f499bc7c479446893f72f8009ef089e in term 0.
I20260924 16:13:28.430825 13783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04992c82ec6345fd801d825cd985d7a3" candidate_uuid: "3f499bc7c479446893f72f8009ef089e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e86c301c70c4f4f9d21b0bb2352361b" is_pre_election: true
I20260924 16:13:28.431110 13783 raft_consensus.cc:2454] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f499bc7c479446893f72f8009ef089e in term 0.
I20260924 16:13:28.431211 13460 leader_election.cc:302] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f499bc7c479446893f72f8009ef089e, a0e4042f6f1548ef803a9f0ad3e20a6b; no voters:
I20260924 16:13:28.431854 13866 raft_consensus.cc:2781] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:13:28.431954 13866 raft_consensus.cc:482] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:13:28.432014 13866 raft_consensus.cc:3037] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:28.433584 13866 raft_consensus.cc:504] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f499bc7c479446893f72f8009ef089e" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38993 } } peers { permanent_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41429 } } peers { permanent_uuid: "4e86c301c70c4f4f9d21b0bb2352361b" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44107 } }
I20260924 16:13:28.434029 13866 leader_election.cc:288] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [CANDIDATE]: Term 1 election: Requested vote from peers a0e4042f6f1548ef803a9f0ad3e20a6b (127.1.253.66:41429), 4e86c301c70c4f4f9d21b0bb2352361b (127.1.253.67:44107)
I20260924 16:13:28.434477 13656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04992c82ec6345fd801d825cd985d7a3" candidate_uuid: "3f499bc7c479446893f72f8009ef089e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b"
I20260924 16:13:28.434607 13656 raft_consensus.cc:3037] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:28.434646 13783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04992c82ec6345fd801d825cd985d7a3" candidate_uuid: "3f499bc7c479446893f72f8009ef089e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e86c301c70c4f4f9d21b0bb2352361b"
I20260924 16:13:28.434777 13783 raft_consensus.cc:3037] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:28.436066 13656 raft_consensus.cc:2454] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f499bc7c479446893f72f8009ef089e in term 1.
I20260924 16:13:28.436586 13460 leader_election.cc:302] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f499bc7c479446893f72f8009ef089e, a0e4042f6f1548ef803a9f0ad3e20a6b; no voters:
I20260924 16:13:28.436690 13783 raft_consensus.cc:2454] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f499bc7c479446893f72f8009ef089e in term 1.
I20260924 16:13:28.436858 13866 raft_consensus.cc:2781] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:28.437065 13866 raft_consensus.cc:686] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [term 1 LEADER]: Becoming Leader. State: Replica: 3f499bc7c479446893f72f8009ef089e, State: Running, Role: LEADER
I20260924 16:13:28.437523 13866 consensus_queue.cc:237] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f499bc7c479446893f72f8009ef089e" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38993 } } peers { permanent_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41429 } } peers { permanent_uuid: "4e86c301c70c4f4f9d21b0bb2352361b" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44107 } }
I20260924 16:13:28.441418 13357 catalog_manager.cc:5495] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f499bc7c479446893f72f8009ef089e (127.1.253.65). New cstate: current_term: 1 leader_uuid: "3f499bc7c479446893f72f8009ef089e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f499bc7c479446893f72f8009ef089e" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e86c301c70c4f4f9d21b0bb2352361b" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44107 } health_report { overall_health: UNKNOWN } } }
W20260924 16:13:28.454811 13710 tablet.cc:2369] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:13:28.496699 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.491161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.1.253.65:38993)
0924 16:13:28.491578 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:28.491583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:28.491596 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:13:28.491763 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:28.491767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:28.491836 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:28.491952 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:13:28.492318 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.493092 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.494389 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.494723 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.494824 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.495249 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:13:28.495844 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:13:28.495965 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:28.496385 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 16:13:28.496447 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333}
I20260924 16:13:28.505896 13783 raft_consensus.cc:1261] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b [term 1 FOLLOWER]: Refusing update from remote peer 3f499bc7c479446893f72f8009ef089e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:28.507026 13656 raft_consensus.cc:1261] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b [term 1 FOLLOWER]: Refusing update from remote peer 3f499bc7c479446893f72f8009ef089e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:28.507891 13873 consensus_queue.cc:1035] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e4042f6f1548ef803a9f0ad3e20a6b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:28.508476 13873 consensus_queue.cc:1035] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e86c301c70c4f4f9d21b0bb2352361b" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:13:28.514374 13572 tablet.cc:2369] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 16:13:28 dist-test-slave-6wvm krb5kdc[12882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266407, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:13:28.531417 13877 mvcc.cc:205] Tried to move back new op lower bound from 7332931209230426112 to 7332931208964317184. Current Snapshot: MvccSnapshot[applied={T|T < 7332931209230426112}]
I20260924 16:13:28.537643 13878 mvcc.cc:205] Tried to move back new op lower bound from 7332931209230426112 to 7332931208964317184. Current Snapshot: MvccSnapshot[applied={T|T < 7332931209230426112}]
I20260924 16:13:28.550894 13880 mvcc.cc:205] Tried to move back new op lower bound from 7332931209230426112 to 7332931208964317184. Current Snapshot: MvccSnapshot[applied={T|T < 7332931209230426112}]
I20260924 16:13:28.553081 13358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:28.553280 13358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:28.556093 13358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e805c25782c94381a79db0dc5031c4c7: Sending DeleteTablet for 3 replicas of tablet 04992c82ec6345fd801d825cd985d7a3
I20260924 16:13:28.557272 13633 tablet_service.cc:1514] Processing DeleteTablet for tablet 04992c82ec6345fd801d825cd985d7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:28 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:59744
I20260924 16:13:28.557300 13765 tablet_service.cc:1514] Processing DeleteTablet for tablet 04992c82ec6345fd801d825cd985d7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:28 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:56040
I20260924 16:13:28.557819 13502 tablet_service.cc:1514] Processing DeleteTablet for tablet 04992c82ec6345fd801d825cd985d7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:28 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:34978
I20260924 16:13:28.558360 13889 tablet_replica.cc:330] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e: stopping tablet replica
I20260924 16:13:28.558671 13888 tablet_replica.cc:330] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b: stopping tablet replica
I20260924 16:13:28.558815 13889 raft_consensus.cc:2229] T 04992c82ec6345fd801d825cd985d7a3 P 3f499bc7c479446893f72f8009ef089e [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:13:28.558983 13888 raft_consensus.cc:2229] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:28.559099 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 13439
I20260924 16:13:28.559955 13887 tablet_replica.cc:330] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b: stopping tablet replica
I20260924 16:13:28.560233 13887 raft_consensus.cc:2229] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:28.560576 13887 raft_consensus.cc:2258] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:28.562505 13888 raft_consensus.cc:2258] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:28.563907 13887 ts_tablet_manager.cc:1905] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:28.566835 13887 ts_tablet_manager.cc:1918] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:28.566943 13887 log.cc:1196] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/04992c82ec6345fd801d825cd985d7a3
I20260924 16:13:28.568449 13888 ts_tablet_manager.cc:1905] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 16:13:28.570616 13344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:38993 (error 108)
I20260924 16:13:28.571099 13888 ts_tablet_manager.cc:1918] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:28.571192 13888 log.cc:1196] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/04992c82ec6345fd801d825cd985d7a3
I20260924 16:13:28.571470 13888 ts_tablet_manager.cc:1939] T 04992c82ec6345fd801d825cd985d7a3 P a0e4042f6f1548ef803a9f0ad3e20a6b: Deleting consensus metadata
I20260924 16:13:28.571694 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 13579
I20260924 16:13:28.572499 13887 ts_tablet_manager.cc:1939] T 04992c82ec6345fd801d825cd985d7a3 P 4e86c301c70c4f4f9d21b0bb2352361b: Deleting consensus metadata
I20260924 16:13:28.574126 13345 catalog_manager.cc:4841] TS 4e86c301c70c4f4f9d21b0bb2352361b (127.1.253.67:44107): tablet 04992c82ec6345fd801d825cd985d7a3 (table test-table [id=e316bdba19864591a4913301810fffba]) successfully deleted
I20260924 16:13:28.575419 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.575036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:38993 (local address 127.0.0.1:34990)
0924 16:13:28.575228 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:28.575311 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 16:13:28.575618 13344 catalog_manager.cc:4584] TS 3f499bc7c479446893f72f8009ef089e (127.1.253.65:38993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04992c82ec6345fd801d825cd985d7a3: Network error: Client connection negotiation failed: client connection to 127.1.253.65:38993: connect: Connection refused (error 111)
I20260924 16:13:28.581540 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 13714
I20260924 16:13:28.589689 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 13326
I20260924 16:13:28.598901 2037 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 16:13:28.944890 2037 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 16:13:28.945024 2037 mini_ranger.cc:66] Stopping Ranger...
I20260924 16:13:29.290877 2037 mini_ranger.cc:68] Stopped Ranger
2026-09-24 16:13:29.291 UTC [12912] LOG: received smart shutdown request
2026-09-24 16:13:29.294 UTC [12912] LOG: background worker "logical replication launcher" (PID 12921) exited with exit code 1
2026-09-24 16:13:29.295 UTC [12916] LOG: shutting down
2026-09-24 16:13:29.482 UTC [12912] LOG: database system is shut down
2026-09-24T16:13:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49646 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:13:29 dist-test-slave-6wvm krb5kdc[13900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:13:29 dist-test-slave-6wvm krb5kdc[13900](info): set up 2 sockets
Sep 24 16:13:29 dist-test-slave-6wvm krb5kdc[13900](info): commencing operation
krb5kdc: starting...
W20260924 16:13:30.646488 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:13:30 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:13:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:13:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.253.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.253.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/rangeradmin_127.1.253.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/rangeradmin_127.1.253.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.253.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.253.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/rangerlookup_127.1.253.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/rangerlookup_127.1.253.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.253.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.253.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/HTTP_127.1.253.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/HTTP_127.1.253.84@KRBTEST.COM.keytab.
I20260924 16:13:30.827391 2037 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6Pu02A/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/postgres -l logfile start
2026-09-24 16:13:33.426 UTC [13930] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 16:13:33.426 UTC [13930] LOG: listening on IPv4 address "127.1.253.84", port 41521
2026-09-24 16:13:33.430 UTC [13930] LOG: listening on Unix socket "/tmp/.s.PGSQL.41521"
2026-09-24 16:13:33.451 UTC [13933] LOG: database system was shut down at 2026-09-24 16:13:31 UTC
2026-09-24 16:13:33.455 UTC [13930] LOG: database system is ready to accept connections
I20260924 16:13:34.426359 2037 mini_postgres.cc:96] Postgres bound to 41521
2026-09-24 16:13:34.433 UTC [13941] FATAL: database "slave" does not exist
127.1.253.84:41521 - accepting connections
I20260924 16:13:34.434028 2037 mini_ranger.cc:161] Starting Ranger...
I20260924 16:13:34.451079 2037 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 16:13:34.961931 2037 mini_ranger.cc:87] Created ranger Postgres database
I20260924 16:13:34.962030 2037 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-admin
2026-09-24 16:13:35,273 [I] DB FLAVOR :POSTGRES
2026-09-24 16:13:35,274 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:13:35,274 [I] Checking connection..
2026-09-24 16:13:35,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 16:13:35,526 [I] Checking connection passed.
2026-09-24 16:13:35,526 [I] --------- Verifying version history table ---------
2026-09-24 16:13:35,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:13:35,795 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:13:35,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:13:36,035 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:13:36,035 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 16:13:36,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 16:13:36.304 UTC [14005] WARNING: there is no transaction in progress
2026-09-24 16:13:36,311 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 16:13:36,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:13:36,556 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 16:13:36,557 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:13:36,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 16:13:36,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 16:13:37,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6wvm.c.gcp-upstream.internal','N') ;"
2026-09-24 16:13:37,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:13:37,575 [I] Table x_portal_user does not exist in database ranger
2026-09-24 16:13:37,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:13:37,816 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 16:13:37,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:13:38,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:13:38,346 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 16:13:38,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 16:13:39.415 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.433 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.451 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.469 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.487 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.507 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.580 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.588 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.603 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.615 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.627 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.642 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.652 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.660 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.668 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:39.677 UTC [14113] WARNING: there is no transaction in progress
2026-09-24 16:13:40,401 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 16:13:40,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 16:13:40,664 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 16:13:40,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:13:40,934 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 16:13:40,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:13:41,179 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 16:13:41,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:13:41,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:13:41,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6wvm.c.gcp-upstream.internal';"
2026-09-24 16:13:41,939 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 16:13:41,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 16:13:42,211 [I] DB_PATCHES have already been applied
I20260924 16:13:42.217991 2037 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-admin:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6Pu02A/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 16:13:42.218078 2037 mini_ranger.cc:193] Using host: 127.1.253.84
I20260924 16:13:42.222469 2037 mini_ranger.cc:227] Ranger admin URL: http://127.1.253.84:50203
Sep 24, 2026 4:13:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:13:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:13:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:13:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:13:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:13:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:13:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 16:13:43.346184 2037 mini_ranger.cc:160] Time spent starting Ranger: real 8.912s user 0.008s sys 0.003s
Sep 24, 2026 4:14:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:14:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 24, 2026 4:14:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:14:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:14:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:14:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 16:14:04.348058 2037 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.253.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.253.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/rangerkms_127.1.253.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.253.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/rangerkms_127.1.253.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.253.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.253.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.253.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/HTTP_127.1.253.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.253.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/HTTP_127.1.253.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 16:14:04 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266444, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 16:14:04.438184 2037 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 16:14:04.448769 2037 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 16:14:05.363862 2037 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 16:14:05.363967 2037 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-kms
I20260924 16:14:05.363993 2037 mini_ranger_kms.cc:227] Using postgres at 127.1.253.84:41521
2026-09-24 16:14:05,541 [I] DB FLAVOR :POSTGRES
2026-09-24 16:14:05,541 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:14:05,541 [I] Checking connection
2026-09-24 16:14:05,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 16:14:05,795 [I] connection success
2026-09-24 16:14:05,796 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 16:14:05,796 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 16:14:05,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 16:14:06,057 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 16:14:06,057 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:14:06,057 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 16:14:06,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.253.84:41521/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 16:14:06,321 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 16:14:06.832072 2037 mini_ranger_kms.cc:310] Created kms service
I20260924 16:14:07.047706 2037 mini_ranger_kms.cc:326] Created kudu user
I20260924 16:14:07.097057 2037 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 16:14:07 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266444, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:14:07.488338 2037 mini_ranger_kms.cc:384] Added ranger policy
I20260924 16:14:07.488504 2037 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-kms:/tmp/dist-test-task6Pu02A/build/debug/bin/postgresql.jar:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6Pu02A/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 16:14:07.488533 2037 mini_ranger_kms.cc:240] Using host: 127.1.253.84
I20260924 16:14:07.492892 2037 mini_ranger_kms.cc:276] Ranger KMS PID: 14293
I20260924 16:14:07.492991 2037 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.253.84:59327
Sep 24, 2026 4:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 16:14:09 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266449, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.253.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:09.753674 2037 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.315s user 0.006s sys 0.156s
I20260924 16:14:09.753818 2037 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 16:14:09.753887 2037 mini_ranger_kms.cc:401] 127.1.253.84:59327/kms/v1/keys
Sep 24 16:14:10 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266449, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.253.84@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
Sep 24 16:14:10 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266449, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.253.84@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
Sep 24 16:14:12 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:14:12.750169 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:45503
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:45503
--ranger_config_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.253.84:59327/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:12.874385 14344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:12.874753 14344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:12.874842 14344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:12.879235 14344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:14:12.879343 14344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:12.879379 14344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:12.879418 14344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:14:12.879454 14344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:14:12.886229 14344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.253.84:59327/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:45503
--ranger_config_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:12.887634 14344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:12.888837 14344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:12.898948 14350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:12.898958 14349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:12.899174 14352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:12.900002 14344 server_base.cc:1034] running on GCE node
I20260924 16:14:12.900626 14344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:12.901862 14344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:12.903055 14344 hybrid_clock.cc:648] HybridClock initialized: now 1790266452903032 us; error 41 us; skew 500 ppm
Sep 24 16:14:12 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266452, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:12.906095 14344 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:14:12.907176 14344 webserver.cc:460] Webserver started at http://127.1.253.126:43131/ using document root <none> and password file <none>
I20260924 16:14:12.907766 14344 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:12.907846 14344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:12.908073 14344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:14:12 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266452, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:14:12.963156 14344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "80d8bc3f1c1c4805aa5b2faf750f6766"
format_stamp: "Formatted at 2026-09-24 16:14:12 on dist-test-slave-6wvm"
server_key: "0b52308008909f16d6b2c54127a17b50"
server_key_iv: "f696c42b9ff6714a5cc80683a9c48037"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:12.963824 14344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "80d8bc3f1c1c4805aa5b2faf750f6766"
format_stamp: "Formatted at 2026-09-24 16:14:12 on dist-test-slave-6wvm"
server_key: "0b52308008909f16d6b2c54127a17b50"
server_key_iv: "f696c42b9ff6714a5cc80683a9c48037"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:12.968120 14344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:14:12.986963 14359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:12.988509 14344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:14:12.988679 14344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "80d8bc3f1c1c4805aa5b2faf750f6766"
format_stamp: "Formatted at 2026-09-24 16:14:12 on dist-test-slave-6wvm"
server_key: "0b52308008909f16d6b2c54127a17b50"
server_key_iv: "f696c42b9ff6714a5cc80683a9c48037"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:12.988839 14344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:13.021381 14344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:13.024895 14344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:13.025120 14344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:13.033120 14344 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:45503
I20260924 16:14:13.033123 14411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:45503 every 8 connection(s)
I20260924 16:14:13.034302 14344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:14:13.037448 14412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:13.039817 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 14344
I20260924 16:14:13.039937 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:14:13.040277 2037 external_mini_cluster.cc:1428] Setting key 21781aaa22bab53cfc98ef6b0d8b517a
I20260924 16:14:13.044502 14412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766: Bootstrap starting.
I20260924 16:14:13.047574 14412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:13.048496 14412 log.cc:824] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:14:13 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:13.051031 14412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766: No bootstrap required, opened a new log
I20260924 16:14:13.055181 14412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45503 } }
I20260924 16:14:13.055544 14412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:13.055644 14412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d8bc3f1c1c4805aa5b2faf750f6766, State: Initialized, Role: FOLLOWER
I20260924 16:14:13.056195 14412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45503 } }
I20260924 16:14:13.056321 14412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:14:13.056427 14412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:14:13.056551 14412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:13.056979 14415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:13.041880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.1.253.126:45503)
0924 16:14:13.042381 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:13.042402 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:13.042448 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:14:13.043297 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:13.043326 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:13.043788 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:13.044191 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:13.045703 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:13.046853 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:13.048072 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:13.048601 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:13.048782 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:13.051820 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:13.051825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:13.051842 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:13.051872 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:13.053698 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:13.054233 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:13.054291 (+ 58us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:13.054297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:13.054413 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:13.054711 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:13.054713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:13.054714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:13.055135 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:13.055288 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:13.055690 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 16:14:13.055997 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":122,"threads_started":1}
I20260924 16:14:13.057658 14412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45503 } }
I20260924 16:14:13.058037 14412 leader_election.cc:302] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80d8bc3f1c1c4805aa5b2faf750f6766; no voters:
I20260924 16:14:13.058435 14412 leader_election.cc:288] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:14:13.058569 14417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:14:13.058805 14417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 1 LEADER]: Becoming Leader. State: Replica: 80d8bc3f1c1c4805aa5b2faf750f6766, State: Running, Role: LEADER
I20260924 16:14:13.059180 14417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45503 } }
I20260924 16:14:13.059751 14412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:14:13.060935 14418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45503 } } }
I20260924 16:14:13.061070 14418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:13.061393 14419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80d8bc3f1c1c4805aa5b2faf750f6766. Latest consensus state: current_term: 1 leader_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45503 } } }
I20260924 16:14:13.061456 14426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:14:13.061491 14419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:13.064458 14426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:14:13.064931 14412 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:14:13.074193 14426 catalog_manager.cc:1347] Generated new cluster ID: fd0350a4244943c89ff94980255e4a92
I20260924 16:14:13.074308 14426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:14:13.082163 14426 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:14:13.083451 14426 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:14:13.097119 14426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766: Generated new TSK 0
I20260924 16:14:13.098044 14426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:14:13.794503 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.253.84:59327/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45503
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:13.927156 14458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:13.927479 14458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:13.927596 14458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:13.931854 14458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:13.931957 14458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:13.932085 14458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:14:13.938216 14458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.253.84:59327/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:13.939494 14458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:13.940718 14458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:13.952091 14463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:13.952106 14464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:13.952554 14458 server_base.cc:1034] running on GCE node
W20260924 16:14:13.952759 14466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:13.953280 14458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:13.954136 14458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:13.957526 14458 hybrid_clock.cc:648] HybridClock initialized: now 1790266453957497 us; error 41 us; skew 500 ppm
Sep 24 16:14:13 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:13.961108 14458 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:14:13.962428 14458 webserver.cc:460] Webserver started at http://127.1.253.65:34843/ using document root <none> and password file <none>
I20260924 16:14:13.963251 14458 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:13.963394 14458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:13.963697 14458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:14:13 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:14:13.979970 14458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "3eec3d5a552945cbb1ad9f27a084d997"
format_stamp: "Formatted at 2026-09-24 16:14:13 on dist-test-slave-6wvm"
server_key: "e00b096fd7c71c6a3ba1a5c94a5c7dec"
server_key_iv: "185249c8d475780b0f4f7a0277d62582"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:13.981065 14458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "3eec3d5a552945cbb1ad9f27a084d997"
format_stamp: "Formatted at 2026-09-24 16:14:13 on dist-test-slave-6wvm"
server_key: "e00b096fd7c71c6a3ba1a5c94a5c7dec"
server_key_iv: "185249c8d475780b0f4f7a0277d62582"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:13.986128 14458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:14:14.001948 14475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:14.003854 14458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:14:14.004098 14458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "3eec3d5a552945cbb1ad9f27a084d997"
format_stamp: "Formatted at 2026-09-24 16:14:13 on dist-test-slave-6wvm"
server_key: "e00b096fd7c71c6a3ba1a5c94a5c7dec"
server_key_iv: "185249c8d475780b0f4f7a0277d62582"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:14.004357 14458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:14.038484 14458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:14.042305 14458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:14.042563 14458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:14.043388 14458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:14.044507 14458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:14.044620 14458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:14.044742 14458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:14.044826 14458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:14.055459 14458 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:41811
I20260924 16:14:14.056675 14458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:14:14.058002 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 14458
I20260924 16:14:14.058122 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:14:14.058490 2037 external_mini_cluster.cc:1428] Setting key ca212345fded3640118b8fe3607657c6
I20260924 16:14:14.067075 14588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:41811 every 8 connection(s)
Sep 24 16:14:14 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:14.098541 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:14.074468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:46157 (local address 127.1.253.126:45503)
0924 16:14:14.075321 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:14.075329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:14.077057 (+ 1728us) server_negotiation.cc:407] Connection header received
0924 16:14:14.077111 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:14.077114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:14.077170 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:14.077259 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:14.078960 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:14.079830 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:14.081667 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:14.082296 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:14.082447 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:14.093454 (+ 11007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:14.093459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:14.093462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:14.093496 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:14.095547 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:14.096300 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:14.096304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:14.096306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:14.096401 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:14.096841 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:14.096845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:14.096846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:14.097104 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:14.097202 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:14.098205 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0924 16:14:14.098366 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":122,"threads_started":1}
I20260924 16:14:14.099316 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:14.073554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45503 (local address 127.1.253.65:46157)
0924 16:14:14.075598 (+ 2044us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:14.075657 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:14.076729 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:14.077593 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:14.077603 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:14.078010 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:14.078764 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:14.078785 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:14.080014 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:14.080017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:14.081526 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:14.081538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:14.082985 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:14.082995 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:14.083909 (+ 914us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:14.085546 (+ 1637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:14.085690 (+ 144us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:14.093194 (+ 7504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:14.095716 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:14.095723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:14.095745 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:14.096162 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:14.096507 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:14.096510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:14.096512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:14.096712 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:14.097262 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:14.097269 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:14.097640 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 16:14:14.097892 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 16:14:14.098322 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1808,"thread_start_us":227,"threads_started":1}
I20260924 16:14:14.100900 14589 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45503
I20260924 16:14:14.101167 14589 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:14.101819 14589 heartbeater.cc:511] Master 127.1.253.126:45503 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:14:14.103847 14376 ts_manager.cc:194] Registered new tserver with Master: 3eec3d5a552945cbb1ad9f27a084d997 (127.1.253.65:41811)
I20260924 16:14:14.105440 14376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:46157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:14:14.142409 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.253.84:59327/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45503
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:14:14 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 16:14:14.313774 14597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:14.314157 14597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:14.314689 14597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:14.321009 14597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:14.321153 14597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:14.321313 14597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:14:14.327813 14597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.253.84:59327/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:14.329288 14597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:14.330461 14597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:14.340200 14606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:14.340293 14603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:14.340215 14604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:14.340958 14597 server_base.cc:1034] running on GCE node
I20260924 16:14:14.341459 14597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:14.342226 14597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:14.343487 14597 hybrid_clock.cc:648] HybridClock initialized: now 1790266454343414 us; error 89 us; skew 500 ppm
Sep 24 16:14:14 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:14.347164 14597 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:14:14.348347 14597 webserver.cc:460] Webserver started at http://127.1.253.66:39961/ using document root <none> and password file <none>
I20260924 16:14:14.349051 14597 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:14.349134 14597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:14.349363 14597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:14:14 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:14:14.369007 14597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "bf00097fd93f4b77aabe69e3a9ea9a16"
format_stamp: "Formatted at 2026-09-24 16:14:14 on dist-test-slave-6wvm"
server_key: "ade903f2a391180c089365e31eb29d4a"
server_key_iv: "d10dbd9de81c33ab45b147fa8e058dcd"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:14.370544 14597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "bf00097fd93f4b77aabe69e3a9ea9a16"
format_stamp: "Formatted at 2026-09-24 16:14:14 on dist-test-slave-6wvm"
server_key: "ade903f2a391180c089365e31eb29d4a"
server_key_iv: "d10dbd9de81c33ab45b147fa8e058dcd"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:14.376556 14597 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 16:14:14.411635 14613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:14.413595 14597 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.000s
I20260924 16:14:14.413829 14597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "bf00097fd93f4b77aabe69e3a9ea9a16"
format_stamp: "Formatted at 2026-09-24 16:14:14 on dist-test-slave-6wvm"
server_key: "ade903f2a391180c089365e31eb29d4a"
server_key_iv: "d10dbd9de81c33ab45b147fa8e058dcd"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:14.414062 14597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:14.441521 14597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:14.445312 14597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:14.445596 14597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:14.446460 14597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:14.447675 14597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:14.447781 14597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:14.447965 14597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:14.448029 14597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:14.462164 14597 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:40391
I20260924 16:14:14.463508 14597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:14:14.464936 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:40391 every 8 connection(s)
I20260924 16:14:14.468410 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 14597
I20260924 16:14:14.468533 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:14:14.468815 2037 external_mini_cluster.cc:1428] Setting key 87c329d889bb322622b94fc93498b760
Sep 24 16:14:14 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:14:14.489542 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:14.467028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:40711 (local address 127.1.253.126:45503)
0924 16:14:14.467162 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:14.467167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:14.469718 (+ 2551us) server_negotiation.cc:407] Connection header received
0924 16:14:14.471038 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:14.471042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:14.471099 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:14.471152 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:14.473124 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:14.473991 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:14.476455 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:14.476822 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:14.476904 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:14.484594 (+ 7690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:14.484601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:14.484607 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:14.484646 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:14.486536 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:14.487317 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:14.487322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:14.487324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:14.487420 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:14.487875 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:14.487881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:14.487886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:14.488158 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:14.488249 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:14.489192 (+ 943us) server_negotiation.cc:299] Negotiation successful
0924 16:14:14.489355 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
I20260924 16:14:14.494124 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:14.467588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45503 (local address 127.1.253.66:40711)
0924 16:14:14.469542 (+ 1954us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:14.469610 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:14.470754 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:14.471600 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:14.471610 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:14.471995 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:14.472932 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:14.472956 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:14.474159 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:14.474163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:14.476302 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:14.476314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:14.478500 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:14.478503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:14.478715 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:14.482471 (+ 3756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:14.482495 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:14.484253 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:14.486738 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:14.486746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:14.486768 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:14.487181 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:14.487541 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:14.487544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:14.487546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:14.487751 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:14.488325 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:14.488333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:14.488713 (+ 380us) client_negotiation.cc:769] Sending connection context
0924 16:14:14.489816 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0924 16:14:14.490396 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1705,"thread_start_us":148,"threads_started":1}
I20260924 16:14:14.495668 14727 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45503
I20260924 16:14:14.495982 14727 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:14.496692 14727 heartbeater.cc:511] Master 127.1.253.126:45503 requested a full tablet report, sending...
I20260924 16:14:14.498253 14376 ts_manager.cc:194] Registered new tserver with Master: bf00097fd93f4b77aabe69e3a9ea9a16 (127.1.253.66:40391)
I20260924 16:14:14.499076 14376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:40711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:14:14.522116 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.253.84:59327/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45503
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:14.670548 14732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:14.670838 14732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:14.670924 14732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:14.675457 14732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:14.675525 14732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:14.675622 14732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:14:14.681859 14732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.253.84:59327/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:14.683035 14732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:14.684155 14732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:14.694799 14740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:14.694955 14737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:14.694799 14738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:14.695987 14732 server_base.cc:1034] running on GCE node
I20260924 16:14:14.696491 14732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:14.697413 14732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:14.698827 14732 hybrid_clock.cc:648] HybridClock initialized: now 1790266454698753 us; error 91 us; skew 500 ppm
Sep 24 16:14:14 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:14.702844 14732 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:14:14.704002 14732 webserver.cc:460] Webserver started at http://127.1.253.67:40567/ using document root <none> and password file <none>
I20260924 16:14:14.704643 14732 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:14.704738 14732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:14.704965 14732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:14:14 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:14:14.724542 14732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "e8f47f27e547475f82d1439ba4c3501a"
format_stamp: "Formatted at 2026-09-24 16:14:14 on dist-test-slave-6wvm"
server_key: "836bb265984a2b625fd5cfe9084d25b1"
server_key_iv: "45cea5ed92ba36873ad3bd113e4e54ab"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:14.725317 14732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "e8f47f27e547475f82d1439ba4c3501a"
format_stamp: "Formatted at 2026-09-24 16:14:14 on dist-test-slave-6wvm"
server_key: "836bb265984a2b625fd5cfe9084d25b1"
server_key_iv: "45cea5ed92ba36873ad3bd113e4e54ab"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:14.731886 14732 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.003s
I20260924 16:14:14.754807 14747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:14.759692 14732 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260924 16:14:14.759883 14732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "e8f47f27e547475f82d1439ba4c3501a"
format_stamp: "Formatted at 2026-09-24 16:14:14 on dist-test-slave-6wvm"
server_key: "836bb265984a2b625fd5cfe9084d25b1"
server_key_iv: "45cea5ed92ba36873ad3bd113e4e54ab"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:14.760061 14732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 24 16:14:14 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:14:14.782037 14732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:14.785887 14732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:14.786154 14732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:14.786931 14732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:14.788025 14732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:14.788141 14732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:14.788242 14732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:14.788300 14732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:14.799782 14732 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:36519
I20260924 16:14:14.801139 14732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:14:14.801592 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 14732
I20260924 16:14:14.801705 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:14:14.802032 2037 external_mini_cluster.cc:1428] Setting key a941984fb260014875ffe5c322670f9b
I20260924 16:14:14.805509 14860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:36519 every 8 connection(s)
Sep 24 16:14:14 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266454, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:14.826452 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:14.810310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:60349 (local address 127.1.253.126:45503)
0924 16:14:14.810467 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:14.810480 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:14.810496 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:14:14.811343 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:14.811347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:14.811409 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:14.811467 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:14.813311 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:14.814149 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:14.816104 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:14.816515 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:14.816600 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:14.820169 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:14.820174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:14.820177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:14.820211 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:14.822173 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:14.822963 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:14.822967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:14.822970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:14.823099 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:14.824854 (+ 1755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:14.824858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:14.824860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:14.825162 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:14.825279 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:14.826078 (+ 799us) server_negotiation.cc:299] Negotiation successful
0924 16:14:14.826229 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:14:14.829629 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:14.809044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45503 (local address 127.1.253.67:60349)
0924 16:14:14.809709 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:14.809774 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:14.811035 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:14.811944 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:14.811955 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:14.812370 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:14.813109 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:14.813130 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:14.814373 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:14.814377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:14.815941 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:14.815953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:14.816853 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:14.816857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:14.817087 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:14.817928 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:14.817952 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:14.819937 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:14.822372 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:14.822380 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:14.822402 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:14.822834 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:14.823238 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:14.823243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:14.823245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:14.824728 (+ 1483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:14.825292 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:14.825300 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:14.825711 (+ 411us) client_negotiation.cc:769] Sending connection context
0924 16:14:14.826017 (+ 306us) client_negotiation.cc:240] Negotiation successful
0924 16:14:14.826444 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":178,"threads_started":1}
I20260924 16:14:14.831063 14861 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45503
I20260924 16:14:14.831337 14861 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:14.831979 14861 heartbeater.cc:511] Master 127.1.253.126:45503 requested a full tablet report, sending...
I20260924 16:14:14.833125 14376 ts_manager.cc:194] Registered new tserver with Master: e8f47f27e547475f82d1439ba4c3501a (127.1.253.67:36519)
I20260924 16:14:14.833925 14376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:60349
I20260924 16:14:14.840128 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:14:14.841061 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 14458
I20260924 16:14:14.847949 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 14597
I20260924 16:14:14.856731 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 14732
I20260924 16:14:14.863977 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 14344
I20260924 16:14:14.872639 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:45503
--webserver_interface=127.1.253.126
--webserver_port=43131
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:45503
--ranger_config_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.253.84:59327/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.253.84:59327/kms with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:14.988004 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:14.988351 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:14.988435 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:14.992636 14864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:14:14.992743 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:14.992781 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:14.992812 14864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:14:14.992846 14864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:14:14.998327 14864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.253.84:59327/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:45503
--ranger_config_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=43131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:14.999527 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:15.000674 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:15.010002 14870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:15.009946 14872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:15.009960 14864 server_base.cc:1034] running on GCE node
W20260924 16:14:15.009971 14869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:15.010793 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:15.012008 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:15.013206 14864 hybrid_clock.cc:648] HybridClock initialized: now 1790266455013190 us; error 28 us; skew 500 ppm
Sep 24 16:14:15 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:15.016194 14864 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:14:15.017165 14864 webserver.cc:460] Webserver started at http://127.1.253.126:43131/ using document root <none> and password file <none>
I20260924 16:14:15.017719 14864 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:15.017786 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:14:15 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:14:15.078591 14864 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:14:15.081228 14880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:15.083292 14864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:14:15.083432 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "80d8bc3f1c1c4805aa5b2faf750f6766"
format_stamp: "Formatted at 2026-09-24 16:14:12 on dist-test-slave-6wvm"
server_key: "0b52308008909f16d6b2c54127a17b50"
server_key_iv: "f696c42b9ff6714a5cc80683a9c48037"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:15.084049 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:15.103173 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:15.106774 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:15.107002 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:15.117229 14864 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:45503
I20260924 16:14:15.117281 14932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:45503 every 8 connection(s)
I20260924 16:14:15.118597 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:14:15.121438 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 14864
I20260924 16:14:15.122030 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.253.84:59327/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:41811
--local_ip_for_outbound_sockets=127.1.253.65
--tserver_master_addrs=127.1.253.126:45503
--webserver_port=34843
--webserver_interface=127.1.253.65
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.253.84:59327/kms with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:14:15.123416 14933 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:14:15.125689 14933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766: Bootstrap starting.
I20260924 16:14:15.142908 14933 log.cc:824] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:15.149914 14933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:14:15.150435 14933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766: Bootstrap complete.
I20260924 16:14:15.154784 14933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45503 } }
I20260924 16:14:15.155223 14933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d8bc3f1c1c4805aa5b2faf750f6766, State: Initialized, Role: FOLLOWER
I20260924 16:14:15.155750 14933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45503 } }
I20260924 16:14:15.155862 14933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:14:15.155951 14933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:14:15.156078 14933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:14:15.158190 14933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45503 } }
I20260924 16:14:15.158609 14933 leader_election.cc:302] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80d8bc3f1c1c4805aa5b2faf750f6766; no voters:
I20260924 16:14:15.158979 14933 leader_election.cc:288] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:14:15.159063 14937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:14:15.159269 14937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [term 2 LEADER]: Becoming Leader. State: Replica: 80d8bc3f1c1c4805aa5b2faf750f6766, State: Running, Role: LEADER
I20260924 16:14:15.159578 14937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45503 } }
I20260924 16:14:15.160266 14933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:14:15.161123 14938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45503 } } }
I20260924 16:14:15.161170 14939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80d8bc3f1c1c4805aa5b2faf750f6766. Latest consensus state: current_term: 2 leader_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d8bc3f1c1c4805aa5b2faf750f6766" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45503 } } }
I20260924 16:14:15.161262 14938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:15.161288 14939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766 [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:15.162799 14933 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:14:15.163187 14945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:14:15.169014 14945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:14:15.169790 14945 catalog_manager.cc:1259] Loaded cluster ID: fd0350a4244943c89ff94980255e4a92
I20260924 16:14:15.169866 14945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:14:15.173251 14945 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:14:15.174268 14945 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766: Loaded TSK: 0
I20260924 16:14:15.175657 14945 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:14:15.271472 14935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:15.271862 14935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:15.271938 14935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:15.277835 14935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:15.277925 14935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:15.278030 14935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:14:15.285789 14935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.253.84:59327/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:41811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=34843
--tserver_master_addrs=127.1.253.126:45503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:15.287263 14935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:15.288658 14935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:15.301761 14971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:15.302289 14935 server_base.cc:1034] running on GCE node
W20260924 16:14:15.302073 14970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:15.302465 14973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:15.303087 14935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:15.303943 14935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:15.305142 14935 hybrid_clock.cc:648] HybridClock initialized: now 1790266455305113 us; error 39 us; skew 500 ppm
Sep 24 16:14:15 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:15.311019 14935 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:14:15.312232 14935 webserver.cc:460] Webserver started at http://127.1.253.65:34843/ using document root <none> and password file <none>
I20260924 16:14:15.312959 14935 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:15.313074 14935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:14:15 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:14:15.346071 14935 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:14:15.350780 14981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:15.352720 14935 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 16:14:15.352946 14935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "3eec3d5a552945cbb1ad9f27a084d997"
format_stamp: "Formatted at 2026-09-24 16:14:13 on dist-test-slave-6wvm"
server_key: "e00b096fd7c71c6a3ba1a5c94a5c7dec"
server_key_iv: "185249c8d475780b0f4f7a0277d62582"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:15.353813 14935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:15.387708 14935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:15.398830 14935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:15.399088 14935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:15.399924 14935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:15.401154 14935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:15.401232 14935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:15.401324 14935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:15.401364 14935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:15.415690 14935 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:41811
I20260924 16:14:15.416383 15094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:41811 every 8 connection(s)
I20260924 16:14:15.417146 14935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:14:15.422909 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 14935
I20260924 16:14:15.423471 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.253.84:59327/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:40391
--local_ip_for_outbound_sockets=127.1.253.66
--tserver_master_addrs=127.1.253.126:45503
--webserver_port=39961
--webserver_interface=127.1.253.66
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.253.84:59327/kms with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:14:15 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:15.440115 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:15.419854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45503 (local address 127.1.253.65:36195)
0924 16:14:15.420385 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:15.420441 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:15.421385 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:15.423375 (+ 1990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:15.423388 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:15.424028 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:15.424794 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:15.424819 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.426900 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.426906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:15.427943 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:15.427955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.429139 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.429144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:15.429384 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:15.430229 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:15.430295 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:15.432367 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:15.435561 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:15.435570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:15.435590 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:15.436034 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:15.436555 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:15.436559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:15.436561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:15.436804 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:15.437850 (+ 1046us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:15.437861 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:15.438274 (+ 413us) client_negotiation.cc:769] Sending connection context
0924 16:14:15.438549 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 16:14:15.438989 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":155,"threads_started":1}
I20260924 16:14:15.441356 15098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:15.419821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:36195 (local address 127.1.253.126:45503)
0924 16:14:15.420569 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:15.420596 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:15.420654 (+ 58us) server_negotiation.cc:407] Connection header received
0924 16:14:15.421883 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:15.421901 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:15.422356 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:15.422716 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:15.425026 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.426719 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.428100 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.428916 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.429129 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:15.432655 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:15.432665 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:15.432687 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:15.432733 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:15.435382 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:15.436214 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:15.436225 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:15.436233 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:15.436411 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:15.436953 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:15.436959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:15.436963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:15.437578 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:15.437729 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:15.438726 (+ 997us) server_negotiation.cc:299] Negotiation successful
0924 16:14:15.439075 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":131,"threads_started":1}
I20260924 16:14:15.444463 15095 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45503
I20260924 16:14:15.444769 15095 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:15.445442 15095 heartbeater.cc:511] Master 127.1.253.126:45503 requested a full tablet report, sending...
W20260924 16:14:15.448074 15095 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:45503 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:14:15.448240 15095 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:15.448453 15095 heartbeater.cc:511] Master 127.1.253.126:45503 requested a full tablet report, sending...
I20260924 16:14:15.456664 15095 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:15.457005 15095 heartbeater.cc:511] Master 127.1.253.126:45503 requested a full tablet report, sending...
W20260924 16:14:15.458175 15095 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:14:15.600455 15099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:15.600914 15099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:15.601014 15099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:15.607515 15099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:15.607611 15099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:15.607729 15099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:14:15.616117 15099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.253.84:59327/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:40391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=39961
--tserver_master_addrs=127.1.253.126:45503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:15.617699 15099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:15.619284 15099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:15.632702 15107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:15.632694 15104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:15.633942 15105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:15.634325 15099 server_base.cc:1034] running on GCE node
I20260924 16:14:15.634865 15099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:15.635730 15099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:15.636916 15099 hybrid_clock.cc:648] HybridClock initialized: now 1790266455636835 us; error 98 us; skew 500 ppm
Sep 24 16:14:15 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:15.640681 15099 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:14:15.641870 15099 webserver.cc:460] Webserver started at http://127.1.253.66:39961/ using document root <none> and password file <none>
I20260924 16:14:15.642670 15099 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:15.642788 15099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:14:15 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:14:15.666770 15099 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:14:15.670343 15115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:15.671993 15099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 16:14:15.672153 15099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "bf00097fd93f4b77aabe69e3a9ea9a16"
format_stamp: "Formatted at 2026-09-24 16:14:14 on dist-test-slave-6wvm"
server_key: "ade903f2a391180c089365e31eb29d4a"
server_key_iv: "d10dbd9de81c33ab45b147fa8e058dcd"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:15.672943 15099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:15.690006 15099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:15.694325 15099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:15.694562 15099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:15.695433 15099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:15.696620 15099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:15.696687 15099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:15.696750 15099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:15.696779 15099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:15.710911 15099 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:40391
I20260924 16:14:15.710952 15229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:40391 every 8 connection(s)
I20260924 16:14:15.712430 15099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:14:15.718513 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 15099
I20260924 16:14:15.718964 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.253.84:59327/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:36519
--local_ip_for_outbound_sockets=127.1.253.67
--tserver_master_addrs=127.1.253.126:45503
--webserver_port=40567
--webserver_interface=127.1.253.67
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.253.84:59327/kms with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:14:15 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:15.731215 15098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:15.714689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:50263 (local address 127.1.253.126:45503)
0924 16:14:15.714856 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:15.714861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:15.715982 (+ 1121us) server_negotiation.cc:407] Connection header received
0924 16:14:15.717479 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:15.717483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:15.717544 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:15.717611 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:15.719301 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.720213 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.722308 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.722723 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.722854 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:15.726206 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:15.726212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:15.726218 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:15.726279 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:15.728210 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:15.728949 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:15.728956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:15.728960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:15.729061 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:15.729558 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:15.729564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:15.729568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:15.729844 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:15.729930 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:15.730807 (+ 877us) server_negotiation.cc:299] Negotiation successful
0924 16:14:15.730966 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:14:15.733098 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:15.715148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45503 (local address 127.1.253.66:50263)
0924 16:14:15.715777 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:15.715848 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:15.717124 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:15.717868 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:15.717878 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:15.718329 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:15.719079 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:15.719101 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.720434 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.720438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:15.722124 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:15.722139 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.722941 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.722949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:15.723226 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:15.723987 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:15.724013 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:15.725972 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:15.728373 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:15.728382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:15.728402 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:15.728810 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:15.729198 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:15.729204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:15.729207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:15.729411 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:15.730026 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:15.730036 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:15.730445 (+ 409us) client_negotiation.cc:769] Sending connection context
0924 16:14:15.731419 (+ 974us) client_negotiation.cc:240] Negotiation successful
0924 16:14:15.731837 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":154,"threads_started":1}
I20260924 16:14:15.734575 15230 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45503
I20260924 16:14:15.734858 15230 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:15.735383 15230 heartbeater.cc:511] Master 127.1.253.126:45503 requested a full tablet report, sending...
W20260924 16:14:15.737540 15230 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:45503 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:14:15.737663 15230 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:15.737823 15230 heartbeater.cc:511] Master 127.1.253.126:45503 requested a full tablet report, sending...
I20260924 16:14:15.738667 15230 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:15.738821 15230 heartbeater.cc:511] Master 127.1.253.126:45503 requested a full tablet report, sending...
W20260924 16:14:15.739532 15230 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:14:15.868285 15233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:15.868599 15233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:15.868686 15233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:15.873308 15233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:15.873417 15233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:15.873528 15233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:14:15.880090 15233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.253.84:59327/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:36519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=40567
--tserver_master_addrs=127.1.253.126:45503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:15.881382 15233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:15.882856 15233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:15.895743 15241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:15.895751 15238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:15.895889 15239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:15.896318 15233 server_base.cc:1034] running on GCE node
I20260924 16:14:15.896843 15233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:15.897626 15233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:15.898844 15233 hybrid_clock.cc:648] HybridClock initialized: now 1790266455898802 us; error 41 us; skew 500 ppm
Sep 24 16:14:15 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:15.902180 15233 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:14:15.903472 15233 webserver.cc:460] Webserver started at http://127.1.253.67:40567/ using document root <none> and password file <none>
I20260924 16:14:15.904198 15233 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:15.904317 15233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:14:15 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:14:15.922737 15233 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:14:15.926322 15249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:15.928104 15233 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:14:15.928248 15233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "e8f47f27e547475f82d1439ba4c3501a"
format_stamp: "Formatted at 2026-09-24 16:14:14 on dist-test-slave-6wvm"
server_key: "836bb265984a2b625fd5cfe9084d25b1"
server_key_iv: "45cea5ed92ba36873ad3bd113e4e54ab"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:15.928984 15233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:15.952390 15233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:15.955921 15233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:15.956154 15233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:15.956881 15233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:15.957862 15233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:15.957933 15233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:15.958009 15233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:15.958051 15233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:15.969667 15233 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:36519
I20260924 16:14:15.969892 15362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:36519 every 8 connection(s)
I20260924 16:14:15.970968 15233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:14:15.980219 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 15233
Sep 24 16:14:15 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:15.989609 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:15.973640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45503 (local address 127.1.253.67:33949)
0924 16:14:15.974321 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:15.974389 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:15.975563 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:15.976468 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:15.976490 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:15.976910 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:15.977706 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:15.977730 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.979011 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.979015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:15.980592 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:15.980605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.981402 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.981407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:15.981639 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:15.982407 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:15.982430 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:15.984217 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:15.986233 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:15.986240 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:15.986277 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:15.986638 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:15.986958 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:15.986962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:15.986964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:15.987129 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:15.987634 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:15.987641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:15.988016 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 16:14:15.988243 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 16:14:15.988661 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":171,"threads_started":1}
I20260924 16:14:15.990525 15098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:15.973665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:33949 (local address 127.1.253.126:45503)
0924 16:14:15.973848 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:15.973854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:15.974530 (+ 676us) server_negotiation.cc:407] Connection header received
0924 16:14:15.975861 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:15.975865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:15.975933 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:15.975998 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:15.977912 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.978757 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.980749 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.981086 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.981179 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:15.984406 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:15.984410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:15.984412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:15.984435 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:15.986096 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:15.986763 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:15.986772 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:15.986774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:15.986849 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:15.987239 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:15.987243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:15.987245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:15.987476 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:15.987570 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:15.990180 (+ 2610us) server_negotiation.cc:299] Negotiation successful
0924 16:14:15.990379 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:14:15.991684 15363 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45503
I20260924 16:14:15.991963 15363 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:15.992573 15363 heartbeater.cc:511] Master 127.1.253.126:45503 requested a full tablet report, sending...
I20260924 16:14:15.994113 14892 ts_manager.cc:194] Registered new tserver with Master: e8f47f27e547475f82d1439ba4c3501a (127.1.253.67:36519)
I20260924 16:14:15.995561 14892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:33949
I20260924 16:14:15.999202 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:15.981004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.1.253.126:45503)
0924 16:14:15.982117 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:15.982122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:15.983320 (+ 1198us) server_negotiation.cc:407] Connection header received
0924 16:14:15.983555 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:15.983560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:15.983633 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:15.983736 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:15.985805 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.986656 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.988509 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:15.988882 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:15.988981 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:15.995379 (+ 6398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:15.995383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:15.995386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:15.995414 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:15.997278 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:15.997795 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:15.997800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:15.997802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:15.997895 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:15.998207 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:15.998210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:15.998212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:15.998453 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:15.998637 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:15.998876 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 16:14:15.999011 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":100,"threads_started":1}
Sep 24 16:14:16 dist-test-slave-6wvm krb5kdc[13900](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266456, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:16.458455 15095 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:16.458802 15095 heartbeater.cc:511] Master 127.1.253.126:45503 requested a full tablet report, sending...
I20260924 16:14:16.459770 14892 ts_manager.cc:194] Registered new tserver with Master: 3eec3d5a552945cbb1ad9f27a084d997 (127.1.253.65:41811)
I20260924 16:14:16.460619 14892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:36195
I20260924 16:14:16.739765 15230 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:16.740101 15230 heartbeater.cc:511] Master 127.1.253.126:45503 requested a full tablet report, sending...
I20260924 16:14:16.742511 14892 ts_manager.cc:194] Registered new tserver with Master: bf00097fd93f4b77aabe69e3a9ea9a16 (127.1.253.66:40391)
I20260924 16:14:16.743290 14892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:50263
I20260924 16:14:16.754457 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:14:16.767735 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:16.756107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.1.253.126:45503)
0924 16:14:16.756406 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:16.756411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:16.756975 (+ 564us) server_negotiation.cc:407] Connection header received
0924 16:14:16.757026 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:16.757028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:16.757081 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:16.757155 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:16.758081 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.758985 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.760199 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.760689 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.760898 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:16.763665 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:16.763669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:16.763672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:16.763702 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:16.765511 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:16.766001 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:16.766005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:16.766007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:16.766097 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:16.766438 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:16.766441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:16.766443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:16.766664 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:16.766794 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:16.767428 (+ 634us) server_negotiation.cc:299] Negotiation successful
0924 16:14:16.767558 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":119,"threads_started":1}
I20260924 16:14:16.772116 14892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
Sep 24 16:14:16 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266456, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
W20260924 16:14:16.776767 14892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:14:16.811797 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:16.793911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:36519 (local address 127.0.0.1:60526)
0924 16:14:16.798439 (+ 4528us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:16.798458 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:16.799717 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:16.801779 (+ 2062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:16.801787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:16.801815 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:16.802154 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:16.802165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.805920 (+ 3755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.805924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:16.807465 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:16.807492 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.810613 (+ 3121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.810617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:16.810746 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:16.810974 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 16:14:16.811102 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 16:14:16.811438 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4453,"thread_start_us":94,"threads_started":1}
I20260924 16:14:16.812314 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:16.794761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.1.253.67:36519)
0924 16:14:16.795051 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:16.795058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:16.799859 (+ 4801us) server_negotiation.cc:407] Connection header received
0924 16:14:16.799975 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:16.799982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:16.801258 (+ 1276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:16.802321 (+ 1063us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:16.802491 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.805771 (+ 3280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.807640 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.810198 (+ 2558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.810333 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:16.810537 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:16.812046 (+ 1509us) server_negotiation.cc:299] Negotiation successful
0924 16:14:16.812165 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260924 16:14:16.813978 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:16.793047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:40391 (local address 127.0.0.1:42612)
0924 16:14:16.794370 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:16.794428 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:16.794567 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:16.809504 (+ 14937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:16.809508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:16.809529 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:16.809627 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:16.809633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.811309 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.811313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:16.812537 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:16.812549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.813622 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.813625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:16.813712 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:16.813737 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:14:16.813798 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 16:14:16.813864 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"thread_start_us":137,"threads_started":1}
I20260924 16:14:16.814694 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:16.793189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.1.253.66:40391)
0924 16:14:16.808936 (+ 15747us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:16.808944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:16.808974 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:14:16.809062 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:16.809068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:16.809327 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:16.809809 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:16.809952 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.811150 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.812687 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.813484 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.814277 (+ 793us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:16.814427 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:16.814478 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 16:14:16.814563 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15605,"thread_start_us":140,"threads_started":1}
I20260924 16:14:16.815939 15297 tablet_service.cc:1467] Processing CreateTablet for tablet 2818ed413969496f87c79de01bffebde (DEFAULT_TABLE table=test-table [id=78b9614eca1c4dc2b1c1f507056125b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:16.817138 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:16.794442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.1.253.65:41811)
0924 16:14:16.794837 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:16.794843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:16.800313 (+ 5470us) server_negotiation.cc:407] Connection header received
0924 16:14:16.800575 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:16.800581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:16.800858 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:16.801318 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:16.803310 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.804666 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.814318 (+ 9652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.815174 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.815305 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:16.815462 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:16.816870 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0924 16:14:16.816993 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":121,"threads_started":1}
I20260924 16:14:16.817142 15297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2818ed413969496f87c79de01bffebde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:16.817358 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:16.794271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:41811 (local address 127.0.0.1:43150)
0924 16:14:16.800201 (+ 5930us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:16.800215 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:16.800386 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:16.802021 (+ 1635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:16.802023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:16.802042 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:16.803158 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:16.803167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.804800 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.804803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:16.813533 (+ 8730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:16.813546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:16.815875 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:16.815879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:16.815988 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:16.816015 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:14:16.816081 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 16:14:16.816150 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5859,"thread_start_us":75,"threads_started":1}
I20260924 16:14:16.817409 15164 tablet_service.cc:1467] Processing CreateTablet for tablet 2818ed413969496f87c79de01bffebde (DEFAULT_TABLE table=test-table [id=78b9614eca1c4dc2b1c1f507056125b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:16.818482 15164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2818ed413969496f87c79de01bffebde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:16.820140 15029 tablet_service.cc:1467] Processing CreateTablet for tablet 2818ed413969496f87c79de01bffebde (DEFAULT_TABLE table=test-table [id=78b9614eca1c4dc2b1c1f507056125b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:16.821179 15029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2818ed413969496f87c79de01bffebde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:16.828338 15396 tablet_bootstrap.cc:492] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16: Bootstrap starting.
I20260924 16:14:16.831036 15396 tablet_bootstrap.cc:654] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:16.831955 15396 log.cc:824] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:16.834542 15396 tablet_bootstrap.cc:492] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16: No bootstrap required, opened a new log
I20260924 16:14:16.834743 15396 ts_tablet_manager.cc:1397] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 16:14:16.839112 15396 raft_consensus.cc:348] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 40391 } } peers { permanent_uuid: "e8f47f27e547475f82d1439ba4c3501a" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36519 } } peers { permanent_uuid: "3eec3d5a552945cbb1ad9f27a084d997" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 41811 } }
I20260924 16:14:16.839520 15396 raft_consensus.cc:374] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:16.839622 15396 raft_consensus.cc:729] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf00097fd93f4b77aabe69e3a9ea9a16, State: Initialized, Role: FOLLOWER
I20260924 16:14:16.840338 15396 consensus_queue.cc:260] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 40391 } } peers { permanent_uuid: "e8f47f27e547475f82d1439ba4c3501a" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36519 } } peers { permanent_uuid: "3eec3d5a552945cbb1ad9f27a084d997" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 41811 } }
I20260924 16:14:16.841951 15396 ts_tablet_manager.cc:1428] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 16:14:16.842486 15230 heartbeater.cc:503] Master 127.1.253.126:45503 was elected leader, sending a full tablet report...
I20260924 16:14:16.846690 15399 tablet_bootstrap.cc:492] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a: Bootstrap starting.
I20260924 16:14:16.848904 15399 tablet_bootstrap.cc:654] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:16.849401 15398 tablet_bootstrap.cc:492] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997: Bootstrap starting.
I20260924 16:14:16.849834 15399 log.cc:824] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:16.852035 15399 tablet_bootstrap.cc:492] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a: No bootstrap required, opened a new log
I20260924 16:14:16.852250 15399 ts_tablet_manager.cc:1397] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:14:16.854326 15398 tablet_bootstrap.cc:654] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:16.857048 15399 raft_consensus.cc:348] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 40391 } } peers { permanent_uuid: "e8f47f27e547475f82d1439ba4c3501a" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36519 } } peers { permanent_uuid: "3eec3d5a552945cbb1ad9f27a084d997" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 41811 } }
I20260924 16:14:16.857354 15399 raft_consensus.cc:374] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:16.857424 15399 raft_consensus.cc:729] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f47f27e547475f82d1439ba4c3501a, State: Initialized, Role: FOLLOWER
I20260924 16:14:16.857532 15398 log.cc:824] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:16.857960 15399 consensus_queue.cc:260] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 40391 } } peers { permanent_uuid: "e8f47f27e547475f82d1439ba4c3501a" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36519 } } peers { permanent_uuid: "3eec3d5a552945cbb1ad9f27a084d997" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 41811 } }
I20260924 16:14:16.859086 15399 ts_tablet_manager.cc:1428] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 16:14:16.859545 15363 heartbeater.cc:503] Master 127.1.253.126:45503 was elected leader, sending a full tablet report...
I20260924 16:14:16.860044 15398 tablet_bootstrap.cc:492] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997: No bootstrap required, opened a new log
I20260924 16:14:16.860236 15398 ts_tablet_manager.cc:1397] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20260924 16:14:16.864333 15398 raft_consensus.cc:348] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 40391 } } peers { permanent_uuid: "e8f47f27e547475f82d1439ba4c3501a" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36519 } } peers { permanent_uuid: "3eec3d5a552945cbb1ad9f27a084d997" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 41811 } }
I20260924 16:14:16.864547 15398 raft_consensus.cc:374] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:16.864578 15398 raft_consensus.cc:729] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eec3d5a552945cbb1ad9f27a084d997, State: Initialized, Role: FOLLOWER
I20260924 16:14:16.865010 15398 consensus_queue.cc:260] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 40391 } } peers { permanent_uuid: "e8f47f27e547475f82d1439ba4c3501a" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36519 } } peers { permanent_uuid: "3eec3d5a552945cbb1ad9f27a084d997" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 41811 } }
I20260924 16:14:16.865880 15398 ts_tablet_manager.cc:1428] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 16:14:16.866262 15095 heartbeater.cc:503] Master 127.1.253.126:45503 was elected leader, sending a full tablet report...
W20260924 16:14:16.919332 15096 tablet.cc:2369] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:14:16.964555 15231 tablet.cc:2369] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:14:16.973055 15364 tablet.cc:2369] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 16:14:17 dist-test-slave-6wvm krb5kdc[13900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266456, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for HTTP/127.1.253.84@KRBTEST.COM
I20260924 16:14:17.183871 15403 raft_consensus.cc:482] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:14:17.184126 15403 raft_consensus.cc:504] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 40391 } } peers { permanent_uuid: "e8f47f27e547475f82d1439ba4c3501a" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36519 } } peers { permanent_uuid: "3eec3d5a552945cbb1ad9f27a084d997" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 41811 } }
I20260924 16:14:17.185612 15403 leader_election.cc:288] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf00097fd93f4b77aabe69e3a9ea9a16 (127.1.253.66:40391), 3eec3d5a552945cbb1ad9f27a084d997 (127.1.253.65:41811)
I20260924 16:14:17.197471 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:17.187313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:35233 (local address 127.1.253.66:40391)
0924 16:14:17.187443 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:17.187449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:17.192668 (+ 5219us) server_negotiation.cc:407] Connection header received
0924 16:14:17.192933 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:17.192937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:17.192992 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:17.193045 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:17.193521 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:17.194301 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:17.196202 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:17.196774 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:17.196889 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:17.196943 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:17.197254 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 16:14:17.197334 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 16:14:17.197870 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:17.186082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:40391 (local address 127.1.253.67:35233)
0924 16:14:17.192550 (+ 6468us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:17.192568 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:17.192771 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:17.193204 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:17.193207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:17.193227 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:17.193335 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:17.193341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:17.194437 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:17.194443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:17.196080 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:17.196087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:17.197029 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:17.197032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:17.197115 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:17.197145 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:14:17.197616 (+ 471us) client_negotiation.cc:240] Negotiation successful
0924 16:14:17.197690 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6367,"thread_start_us":157,"threads_started":1}
I20260924 16:14:17.198928 15166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818ed413969496f87c79de01bffebde" candidate_uuid: "e8f47f27e547475f82d1439ba4c3501a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16" is_pre_election: true
I20260924 16:14:17.199275 15166 raft_consensus.cc:2454] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f47f27e547475f82d1439ba4c3501a in term 0.
I20260924 16:14:17.199932 15250 leader_election.cc:302] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf00097fd93f4b77aabe69e3a9ea9a16, e8f47f27e547475f82d1439ba4c3501a; no voters:
I20260924 16:14:17.200242 15403 raft_consensus.cc:2781] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:14:17.200321 15403 raft_consensus.cc:482] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:14:17.200359 15403 raft_consensus.cc:3037] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:17.201699 15403 raft_consensus.cc:504] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 40391 } } peers { permanent_uuid: "e8f47f27e547475f82d1439ba4c3501a" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36519 } } peers { permanent_uuid: "3eec3d5a552945cbb1ad9f27a084d997" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 41811 } }
I20260924 16:14:17.202075 15403 leader_election.cc:288] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [CANDIDATE]: Term 1 election: Requested vote from peers bf00097fd93f4b77aabe69e3a9ea9a16 (127.1.253.66:40391), 3eec3d5a552945cbb1ad9f27a084d997 (127.1.253.65:41811)
I20260924 16:14:17.202759 15166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818ed413969496f87c79de01bffebde" candidate_uuid: "e8f47f27e547475f82d1439ba4c3501a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16"
I20260924 16:14:17.202879 15166 raft_consensus.cc:3037] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:17.204130 15166 raft_consensus.cc:2454] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f47f27e547475f82d1439ba4c3501a in term 1.
I20260924 16:14:17.204514 15250 leader_election.cc:302] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf00097fd93f4b77aabe69e3a9ea9a16, e8f47f27e547475f82d1439ba4c3501a; no voters:
I20260924 16:14:17.204730 15403 raft_consensus.cc:2781] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:14:17.204939 15403 raft_consensus.cc:686] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 1 LEADER]: Becoming Leader. State: Replica: e8f47f27e547475f82d1439ba4c3501a, State: Running, Role: LEADER
I20260924 16:14:17.205317 15403 consensus_queue.cc:237] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 40391 } } peers { permanent_uuid: "e8f47f27e547475f82d1439ba4c3501a" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36519 } } peers { permanent_uuid: "3eec3d5a552945cbb1ad9f27a084d997" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 41811 } }
I20260924 16:14:17.208894 14892 catalog_manager.cc:5495] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f47f27e547475f82d1439ba4c3501a (127.1.253.67). New cstate: current_term: 1 leader_uuid: "e8f47f27e547475f82d1439ba4c3501a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 40391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8f47f27e547475f82d1439ba4c3501a" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3eec3d5a552945cbb1ad9f27a084d997" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 41811 } health_report { overall_health: UNKNOWN } } }
I20260924 16:14:17.212872 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:17.187474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:40621 (local address 127.1.253.65:41811)
0924 16:14:17.187625 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:17.187630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:17.194331 (+ 6701us) server_negotiation.cc:407] Connection header received
0924 16:14:17.194591 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:17.194595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:17.194646 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:17.194694 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:17.195091 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:17.195841 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:17.197203 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:17.197737 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:17.197835 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:17.197892 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:17.212617 (+ 14725us) server_negotiation.cc:299] Negotiation successful
0924 16:14:17.212732 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:14:17.213491 15407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:17.186566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:41811 (local address 127.1.253.67:40621)
0924 16:14:17.194207 (+ 7641us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:17.194221 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:17.194409 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:17.194831 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:17.194834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:17.194850 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:17.194966 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:17.194972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:17.195971 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:17.195974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:17.197076 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:17.197082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:17.212293 (+ 15211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:17.212296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:17.212436 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:17.212466 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:14:17.213226 (+ 760us) client_negotiation.cc:240] Negotiation successful
0924 16:14:17.213338 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7573,"thread_start_us":94,"threads_started":1}
I20260924 16:14:17.214496 15040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818ed413969496f87c79de01bffebde" candidate_uuid: "e8f47f27e547475f82d1439ba4c3501a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eec3d5a552945cbb1ad9f27a084d997" is_pre_election: true
I20260924 16:14:17.214809 15040 raft_consensus.cc:2454] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f47f27e547475f82d1439ba4c3501a in term 0.
I20260924 16:14:17.215132 15039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818ed413969496f87c79de01bffebde" candidate_uuid: "e8f47f27e547475f82d1439ba4c3501a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eec3d5a552945cbb1ad9f27a084d997"
I20260924 16:14:17.215226 15039 raft_consensus.cc:3037] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:17.216584 15039 raft_consensus.cc:2454] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f47f27e547475f82d1439ba4c3501a in term 1.
I20260924 16:14:17.252992 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:17.248594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.1.253.67:36519)
0924 16:14:17.248725 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:17.248730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:17.248823 (+ 93us) server_negotiation.cc:407] Connection header received
0924 16:14:17.249007 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:17.249010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:17.249069 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:17.249185 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:14:17.249454 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:17.249971 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:17.251224 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:17.251566 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:17.251671 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:17.252108 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:14:17.252556 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:14:17.252699 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:17.252761 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 16:14:17.252819 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:14:17.259764 15166 raft_consensus.cc:1261] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16 [term 1 FOLLOWER]: Refusing update from remote peer e8f47f27e547475f82d1439ba4c3501a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:14:17.259764 15039 raft_consensus.cc:1261] T 2818ed413969496f87c79de01bffebde P 3eec3d5a552945cbb1ad9f27a084d997 [term 1 FOLLOWER]: Refusing update from remote peer e8f47f27e547475f82d1439ba4c3501a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:14:17.260664 15409 consensus_queue.cc:1035] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf00097fd93f4b77aabe69e3a9ea9a16" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 40391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:14:17.261220 15409 consensus_queue.cc:1035] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eec3d5a552945cbb1ad9f27a084d997" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 41811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:14:17.277406 15413 mvcc.cc:205] Tried to move back new op lower bound from 7332931408927444992 to 7332931408716734464. Current Snapshot: MvccSnapshot[applied={T|T < 7332931408927444992}]
I20260924 16:14:17.277690 15414 mvcc.cc:205] Tried to move back new op lower bound from 7332931408927444992 to 7332931408716734464. Current Snapshot: MvccSnapshot[applied={T|T < 7332931408927444992 or (T in {7332931408927444992})}]
I20260924 16:14:17.288942 14892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:14:17.289156 14892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:14:17.291764 14892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80d8bc3f1c1c4805aa5b2faf750f6766: Sending DeleteTablet for 3 replicas of tablet 2818ed413969496f87c79de01bffebde
I20260924 16:14:17.293467 15164 tablet_service.cc:1514] Processing DeleteTablet for tablet 2818ed413969496f87c79de01bffebde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:17 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:42612
I20260924 16:14:17.293426 15029 tablet_service.cc:1514] Processing DeleteTablet for tablet 2818ed413969496f87c79de01bffebde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:17 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:43150
I20260924 16:14:17.293607 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 14935
I20260924 16:14:17.293974 15297 tablet_service.cc:1514] Processing DeleteTablet for tablet 2818ed413969496f87c79de01bffebde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:17 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:60526
I20260924 16:14:17.294551 15424 tablet_replica.cc:330] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16: stopping tablet replica
I20260924 16:14:17.294903 15424 raft_consensus.cc:2229] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:14:17.295177 15424 raft_consensus.cc:2258] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:17.296481 15424 ts_tablet_manager.cc:1905] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:14:17.299048 15424 ts_tablet_manager.cc:1918] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:14:17.299234 15424 log.cc:1196] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/2818ed413969496f87c79de01bffebde
I20260924 16:14:17.299702 15424 ts_tablet_manager.cc:1939] T 2818ed413969496f87c79de01bffebde P bf00097fd93f4b77aabe69e3a9ea9a16: Deleting consensus metadata
I20260924 16:14:17.301147 14881 catalog_manager.cc:4841] TS bf00097fd93f4b77aabe69e3a9ea9a16 (127.1.253.66:40391): tablet 2818ed413969496f87c79de01bffebde (table test-table [id=78b9614eca1c4dc2b1c1f507056125b9]) successfully deleted
W20260924 16:14:17.305410 14882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:41811 (error 108)
I20260924 16:14:17.306449 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 15099
I20260924 16:14:17.307356 15425 tablet_replica.cc:330] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a: stopping tablet replica
I20260924 16:14:17.307689 15425 raft_consensus.cc:2229] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:14:17.308154 15425 raft_consensus.cc:2258] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:17.310398 15425 ts_tablet_manager.cc:1905] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:14:17.311739 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:17.311372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:41811 (local address 127.0.0.1:43154)
0924 16:14:17.311575 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:17.311644 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:41811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260924 16:14:17.313752 14882 catalog_manager.cc:4584] TS 3eec3d5a552945cbb1ad9f27a084d997 (127.1.253.65:41811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2818ed413969496f87c79de01bffebde: Network error: Client connection negotiation failed: client connection to 127.1.253.65:41811: connect: Connection refused (error 111)
I20260924 16:14:17.313804 15425 ts_tablet_manager.cc:1918] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:14:17.313899 15425 log.cc:1196] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/2818ed413969496f87c79de01bffebde
I20260924 16:14:17.314227 15425 ts_tablet_manager.cc:1939] T 2818ed413969496f87c79de01bffebde P e8f47f27e547475f82d1439ba4c3501a: Deleting consensus metadata
I20260924 16:14:17.316401 14882 catalog_manager.cc:4841] TS e8f47f27e547475f82d1439ba4c3501a (127.1.253.67:36519): tablet 2818ed413969496f87c79de01bffebde (table test-table [id=78b9614eca1c4dc2b1c1f507056125b9]) successfully deleted
I20260924 16:14:17.321637 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 15233
I20260924 16:14:17.329912 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 14864
I20260924 16:14:17.338886 2037 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 16:14:17.674288 2037 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 16:14:17.674420 2037 mini_ranger.cc:66] Stopping Ranger...
I20260924 16:14:18.020006 2037 mini_ranger.cc:68] Stopped Ranger
2026-09-24 16:14:18.020 UTC [13930] LOG: received smart shutdown request
2026-09-24 16:14:18.024 UTC [13930] LOG: background worker "logical replication launcher" (PID 13939) exited with exit code 1
2026-09-24 16:14:18.024 UTC [13934] LOG: shutting down
2026-09-24 16:14:18.195 UTC [13930] LOG: database system is shut down
2026-09-24T16:14:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48729 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:14:18 dist-test-slave-6wvm krb5kdc[15440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:14:18 dist-test-slave-6wvm krb5kdc[15440](info): set up 2 sockets
Sep 24 16:14:18 dist-test-slave-6wvm krb5kdc[15440](info): commencing operation
krb5kdc: starting...
W20260924 16:14:19.381922 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:14:19 dist-test-slave-6wvm krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:14:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:14:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:14:19.510349 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:34395
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:34395,127.1.253.125:43443,127.1.253.124:38323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:19.626482 15453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:19.626807 15453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:19.626879 15453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:19.630923 15453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:14:19.631017 15453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:19.631058 15453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:19.631117 15453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:14:19.631172 15453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:14:19.636551 15453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:34395,127.1.253.125:43443,127.1.253.124:38323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:34395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:19.637696 15453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:19.638876 15453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:19.647615 15459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:19.647653 15458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:19.647816 15453 server_base.cc:1034] running on GCE node
W20260924 16:14:19.647670 15461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:19.648460 15453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:19.649614 15453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:19.650833 15453 hybrid_clock.cc:648] HybridClock initialized: now 1790266459650811 us; error 32 us; skew 500 ppm
Sep 24 16:14:19 dist-test-slave-6wvm krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:19.659435 15453 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:14:19.660445 15453 webserver.cc:460] Webserver started at http://127.1.253.126:34135/ using document root <none> and password file <none>
I20260924 16:14:19.661010 15453 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:19.661070 15453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:19.661242 15453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:14:19.663405 15453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "52a5d707d6a3453f868959cb8815b328"
format_stamp: "Formatted at 2026-09-24 16:14:19 on dist-test-slave-6wvm"
server_key: "a8786614a24fbf9a0115d8b2a44d73e7"
server_key_iv: "34873bacd0caff6fe10f2f3a6101809c"
server_key_version: "encryptionkey@0"
I20260924 16:14:19.663957 15453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "52a5d707d6a3453f868959cb8815b328"
format_stamp: "Formatted at 2026-09-24 16:14:19 on dist-test-slave-6wvm"
server_key: "a8786614a24fbf9a0115d8b2a44d73e7"
server_key_iv: "34873bacd0caff6fe10f2f3a6101809c"
server_key_version: "encryptionkey@0"
I20260924 16:14:19.667886 15453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:14:19.670463 15468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:19.671764 15453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:14:19.671908 15453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "52a5d707d6a3453f868959cb8815b328"
format_stamp: "Formatted at 2026-09-24 16:14:19 on dist-test-slave-6wvm"
server_key: "a8786614a24fbf9a0115d8b2a44d73e7"
server_key_iv: "34873bacd0caff6fe10f2f3a6101809c"
server_key_version: "encryptionkey@0"
I20260924 16:14:19.672020 15453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:19.690861 15453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:19.694633 15453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:19.694859 15453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:19.702615 15453 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:34395
I20260924 16:14:19.702634 15520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:34395 every 8 connection(s)
I20260924 16:14:19.703646 15453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:14:19.706789 15521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:19.708027 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 15453
I20260924 16:14:19.708174 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:14:19.708526 2037 external_mini_cluster.cc:1428] Setting key 82524c3e886595b02b3ff2988e6759cd
I20260924 16:14:19.711907 15521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.253.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.125@KRBTEST.COM" created.
Sep 24 16:14:19 dist-test-slave-6wvm krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:19.729045 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.714835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:34395 (local address 127.0.0.1:48254)
0924 16:14:19.715402 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.715447 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:19.716372 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:19.717096 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:19.717101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:19.717535 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:19.717802 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:19.717817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.719080 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.719083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:19.719886 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:19.719895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.720712 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.720719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:19.720868 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:19.721543 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:19.721570 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:19.723202 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:19.725845 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:19.725855 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:19.725864 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:19.726118 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:19.726575 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:19.726577 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:19.726579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:19.726700 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:19.727578 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:19.727584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:19.727722 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 16:14:19.727872 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 16:14:19.728167 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":161,"threads_started":1}
I20260924 16:14:19.729053 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.715942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.1.253.126:34395)
0924 16:14:19.716285 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:19.716298 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:19.716347 (+ 49us) server_negotiation.cc:407] Connection header received
0924 16:14:19.716704 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:19.716708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:19.716894 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:19.717099 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:19.717956 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.718937 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.720025 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.720522 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.720662 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:19.723440 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:19.723451 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:19.723471 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:19.723507 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:19.725701 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:19.726277 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:19.726286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:19.726292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:19.726463 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:19.726808 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:19.726822 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:19.726826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:19.727455 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:19.727620 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:19.728002 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 16:14:19.728284 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":220,"thread_start_us":112,"threads_started":1}
I20260924 16:14:19.731900 15521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:14:19.732748 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.732468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:43443 (local address 127.0.0.1:45980)
0924 16:14:19.732608 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.732681 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 16:14:19.733032 15471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.253.125:43443: connect: Connection refused (error 111)
I20260924 16:14:19.734002 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.733772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:43443 (local address 127.0.0.1:45986)
0924 16:14:19.733896 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.733939 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 16:14:19.734207 15521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.125:43443: Network error: Client connection negotiation failed: client connection to 127.1.253.125:43443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.125.keytab.
Entry for principal kudu/127.1.253.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.125.keytab.
I20260924 16:14:19.739210 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.125:43443
--webserver_interface=127.1.253.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.125.keytab
--principal=kudu/127.1.253.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:34395,127.1.253.125:43443,127.1.253.124:38323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:14:19.783550 15521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } attempt: 1
I20260924 16:14:19.784512 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.784176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:43443 (local address 127.0.0.1:45990)
0924 16:14:19.784360 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.784430 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 16:14:19.785296 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.785085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:43443 (local address 127.0.0.1:45992)
0924 16:14:19.785189 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.785232 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 16:14:19.785444 15521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.125:43443: Network error: Client connection negotiation failed: client connection to 127.1.253.125:43443: connect: Connection refused (error 111)
I20260924 16:14:19.853647 15521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } attempt: 2
I20260924 16:14:19.854661 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.854323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:43443 (local address 127.0.0.1:46008)
0924 16:14:19.854507 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.854576 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 16:14:19.855435 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.855231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:43443 (local address 127.0.0.1:46014)
0924 16:14:19.855336 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.855381 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 16:14:19.855581 15521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.125:43443: Network error: Client connection negotiation failed: client connection to 127.1.253.125:43443: connect: Connection refused (error 111)
W20260924 16:14:19.855901 15528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:19.856180 15528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:19.856299 15528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:19.860570 15528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:14:19.860627 15528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:19.860688 15528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:19.860716 15528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:14:19.860736 15528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:14:19.866415 15528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:34395,127.1.253.125:43443,127.1.253.124:38323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.125.keytab
--principal=kudu/127.1.253.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.125:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.253.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:19.867573 15528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:19.868636 15528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:19.878996 15536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:19.878968 15533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:19.879109 15534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:19.879415 15528 server_base.cc:1034] running on GCE node
I20260924 16:14:19.880074 15528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:19.881296 15528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:19.882462 15528 hybrid_clock.cc:648] HybridClock initialized: now 1790266459882445 us; error 31 us; skew 500 ppm
Sep 24 16:14:19 dist-test-slave-6wvm krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:19.885404 15528 init.cc:377] Logged in from keytab as kudu/127.1.253.125@KRBTEST.COM (short username kudu)
I20260924 16:14:19.886487 15528 webserver.cc:460] Webserver started at http://127.1.253.125:38791/ using document root <none> and password file <none>
I20260924 16:14:19.887104 15528 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:19.887169 15528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:19.887395 15528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:14:19.889571 15528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/data/instance:
uuid: "3c6fa4b44a694d63811ac719ec983b4e"
format_stamp: "Formatted at 2026-09-24 16:14:19 on dist-test-slave-6wvm"
server_key: "b9930a3c3db70ccbebcfa4215749077c"
server_key_iv: "65a11e5fbbb2cc0458d642611aa84401"
server_key_version: "encryptionkey@0"
I20260924 16:14:19.890192 15528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/wal/instance:
uuid: "3c6fa4b44a694d63811ac719ec983b4e"
format_stamp: "Formatted at 2026-09-24 16:14:19 on dist-test-slave-6wvm"
server_key: "b9930a3c3db70ccbebcfa4215749077c"
server_key_iv: "65a11e5fbbb2cc0458d642611aa84401"
server_key_version: "encryptionkey@0"
I20260924 16:14:19.894452 15528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:14:19.897202 15543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:19.898744 15528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:14:19.898890 15528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/wal
uuid: "3c6fa4b44a694d63811ac719ec983b4e"
format_stamp: "Formatted at 2026-09-24 16:14:19 on dist-test-slave-6wvm"
server_key: "b9930a3c3db70ccbebcfa4215749077c"
server_key_iv: "65a11e5fbbb2cc0458d642611aa84401"
server_key_version: "encryptionkey@0"
I20260924 16:14:19.899029 15528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:19.919749 15528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:19.923131 15528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:19.923363 15528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:19.931074 15528 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.125:43443
I20260924 16:14:19.931097 15595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.125:43443 every 8 connection(s)
I20260924 16:14:19.932091 15528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
I20260924 16:14:19.934994 15596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:19.936702 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 15528
I20260924 16:14:19.936860 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-1/wal/instance
I20260924 16:14:19.937170 2037 external_mini_cluster.cc:1428] Setting key 93b92016179d26e1c1e58e0b7d632d56
I20260924 16:14:19.940156 15596 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.253.124@KRBTEST.COM; defaulting to no policy
I20260924 16:14:19.946775 15521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.124@KRBTEST.COM" created.
Sep 24 16:14:19 dist-test-slave-6wvm krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.125@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
Sep 24 16:14:19 dist-test-slave-6wvm krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.125@KRBTEST.COM
I20260924 16:14:19.956672 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.942395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.1.253.126:34395)
0924 16:14:19.942544 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:19.942548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:19.943722 (+ 1174us) server_negotiation.cc:407] Connection header received
0924 16:14:19.944696 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:19.944699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:19.944769 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:19.944899 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:19.946110 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.946690 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.948295 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.948609 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.948672 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:19.951625 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:19.951631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:19.951634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:19.951662 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:19.954101 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:19.954718 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:19.954723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:19.954725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:19.954819 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:19.955307 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:19.955313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:19.955318 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:19.955592 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:19.955702 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:19.956309 (+ 607us) server_negotiation.cc:299] Negotiation successful
0924 16:14:19.956448 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":53}
I20260924 16:14:19.957408 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.942947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:34395 (local address 127.0.0.1:48256)
0924 16:14:19.943549 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.943595 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:19.944465 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:19.945073 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:19.945081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:19.945426 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:19.945965 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:19.945978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.946867 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.946870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:19.948160 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:19.948172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.948888 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.948892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:19.949090 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:19.949791 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:19.949811 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:19.951454 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:19.954231 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:19.954258 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:19.954280 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:19.954600 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:19.954941 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:19.954946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:19.954952 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:19.955169 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:19.955710 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:19.955720 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:19.956018 (+ 298us) client_negotiation.cc:769] Sending connection context
0924 16:14:19.956184 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 16:14:19.956445 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":163,"threads_started":1}
I20260924 16:14:19.959187 15596 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:14:19.960808 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.947232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:43443 (local address 127.0.0.1:46028)
0924 16:14:19.947397 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.947412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:19.947516 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:19.948271 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:19.948275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:19.948483 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:19.948690 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:19.948702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.950069 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.950072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:19.950925 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:19.950942 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.951736 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.951739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:19.951867 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:19.952380 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:19.952396 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:19.954093 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:19.958861 (+ 4768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:19.958864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:19.958866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:19.959127 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:19.959476 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:19.959479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:19.959482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:19.959601 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:19.960189 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:19.960195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:19.960332 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 16:14:19.960464 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 16:14:19.960633 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":36}
I20260924 16:14:19.960894 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.947387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.1.253.125:43443)
0924 16:14:19.947763 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:19.947774 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:19.947809 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:14:19.947850 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:19.947855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:19.948041 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:19.948257 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:19.948832 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.949942 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.951091 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.951622 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.951733 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:19.954768 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:19.954781 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:19.954798 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:19.954821 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:19.958728 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:19.959251 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:19.959257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:19.959258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:19.959373 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:19.959728 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:19.959732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:19.959733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:19.960077 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:19.960202 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:19.960541 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 16:14:19.960691 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":172,"threads_started":1}
I20260924 16:14:19.962944 15521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:14:19.963734 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.963477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34012)
0924 16:14:19.963588 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.963639 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 16:14:19.964596 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.964289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34020)
0924 16:14:19.964478 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.964525 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260924 16:14:19.964744 15521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:38323: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.124.keytab.
Entry for principal kudu/127.1.253.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.124.keytab.
Sep 24 16:14:19 dist-test-slave-6wvm krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.125@KRBTEST.COM for kudu/127.1.253.125@KRBTEST.COM
I20260924 16:14:19.966130 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.124:38323
--webserver_interface=127.1.253.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.124.keytab
--principal=kudu/127.1.253.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:34395,127.1.253.125:43443,127.1.253.124:38323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:14:19.969579 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.959830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:43443 (local address 127.0.0.1:46032)
0924 16:14:19.959961 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.959976 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:19.960077 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:19.960690 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:19.960693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:19.960845 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:19.961027 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:19.961035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.962018 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.962022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:19.962837 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:19.962847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.963430 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.963434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:19.963539 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:19.964169 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:19.964189 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:19.965639 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:19.968074 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:19.968077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:19.968079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:19.968273 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:19.968571 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:19.968575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:19.968583 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:19.968666 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:19.969051 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:19.969054 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:19.969144 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 16:14:19.969254 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 16:14:19.969406 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 16:14:19.969619 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.959831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.1.253.125:43443)
0924 16:14:19.960440 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:19.960453 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:19.960469 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:14:19.960521 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:19.960523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:19.960565 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:19.960652 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:19.961150 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.961911 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.962966 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:19.963318 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:19.963405 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:19.965813 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:19.965816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:19.965818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:19.965839 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:19.967941 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:19.968380 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:19.968384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:19.968386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:19.968464 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:19.968762 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:19.968765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:19.968767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:19.968953 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:19.969063 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:19.969310 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 16:14:19.969467 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":201,"thread_start_us":115,"threads_started":1}
I20260924 16:14:19.970618 15596 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:14:19.971474 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.971101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34026)
0924 16:14:19.971256 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.971376 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 16:14:19.971731 15544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
I20260924 16:14:19.972720 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.972430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34042)
0924 16:14:19.972570 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.972635 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:14:19.972904 15596 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:38323: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
I20260924 16:14:19.995931 15521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } attempt: 1
I20260924 16:14:19.996985 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.996602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34044)
0924 16:14:19.996802 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.996883 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 16:14:19.997848 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:19.997595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34058)
0924 16:14:19.997722 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:19.997778 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 16:14:19.998056 15521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:38323: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
I20260924 16:14:20.022271 15596 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } attempt: 1
I20260924 16:14:20.023321 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.022908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34070)
0924 16:14:20.023125 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.023210 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 16:14:20.024143 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.023894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34086)
0924 16:14:20.024023 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.024076 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:14:20.024310 15596 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:38323: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
I20260924 16:14:20.073307 15521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } attempt: 2
I20260924 16:14:20.074309 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.073890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34100)
0924 16:14:20.074079 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.074158 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 16:14:20.075590 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.075296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34104)
0924 16:14:20.075465 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.075522 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 16:14:20.075752 15521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:38323: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
W20260924 16:14:20.089936 15603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:20.090402 15603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:20.090471 15603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 16:14:20.092479 15596 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } attempt: 2
I20260924 16:14:20.093403 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.093047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34120)
0924 16:14:20.093220 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.093294 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 16:14:20.094125 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.093911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34126)
0924 16:14:20.094015 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.094063 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 16:14:20.094353 15596 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:38323: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
W20260924 16:14:20.096084 15603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:14:20.096141 15603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:20.096187 15603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:20.096225 15603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:14:20.096261 15603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:14:20.102090 15603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:34395,127.1.253.125:43443,127.1.253.124:38323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.124.keytab
--principal=kudu/127.1.253.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.124:38323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.253.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:20.103407 15603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:20.104604 15603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:20.113621 15612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:20.113633 15610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:20.113633 15609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:20.113763 15603 server_base.cc:1034] running on GCE node
I20260924 16:14:20.114725 15603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:20.115957 15603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:20.117172 15603 hybrid_clock.cc:648] HybridClock initialized: now 1790266460117142 us; error 42 us; skew 500 ppm
Sep 24 16:14:20 dist-test-slave-6wvm krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:20.120491 15603 init.cc:377] Logged in from keytab as kudu/127.1.253.124@KRBTEST.COM (short username kudu)
I20260924 16:14:20.121466 15603 webserver.cc:460] Webserver started at http://127.1.253.124:35617/ using document root <none> and password file <none>
I20260924 16:14:20.122061 15603 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:20.122124 15603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:20.122434 15603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:14:20.124557 15603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/data/instance:
uuid: "f44943d2273a4634afe254561539f157"
format_stamp: "Formatted at 2026-09-24 16:14:20 on dist-test-slave-6wvm"
server_key: "00220cc2f6401a2d5f1c36d59635848e"
server_key_iv: "bc9db00e45a02d41eae411415524a193"
server_key_version: "encryptionkey@0"
I20260924 16:14:20.125173 15603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/wal/instance:
uuid: "f44943d2273a4634afe254561539f157"
format_stamp: "Formatted at 2026-09-24 16:14:20 on dist-test-slave-6wvm"
server_key: "00220cc2f6401a2d5f1c36d59635848e"
server_key_iv: "bc9db00e45a02d41eae411415524a193"
server_key_version: "encryptionkey@0"
I20260924 16:14:20.129035 15603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 16:14:20.131534 15619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:20.132745 15603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:14:20.132896 15603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/wal
uuid: "f44943d2273a4634afe254561539f157"
format_stamp: "Formatted at 2026-09-24 16:14:20 on dist-test-slave-6wvm"
server_key: "00220cc2f6401a2d5f1c36d59635848e"
server_key_iv: "bc9db00e45a02d41eae411415524a193"
server_key_version: "encryptionkey@0"
I20260924 16:14:20.133013 15603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:20.166112 15603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:20.169540 15603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:20.169823 15603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:20.174911 15521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } attempt: 3
I20260924 16:14:20.175866 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.175533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34130)
0924 16:14:20.175687 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.175762 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 16:14:20.176755 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.176502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34136)
0924 16:14:20.176622 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.176680 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:14:20.176966 15521 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:38323: Network error: Client connection negotiation failed: client connection to 127.1.253.124:38323: connect: Connection refused (error 111)
I20260924 16:14:20.178085 15603 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.124:38323
I20260924 16:14:20.178097 15671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.124:38323 every 8 connection(s)
I20260924 16:14:20.179162 15603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
I20260924 16:14:20.182127 15672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:20.183758 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 15603
I20260924 16:14:20.183917 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266297063660-2037-0/minicluster-data/master-2/wal/instance
I20260924 16:14:20.184209 2037 external_mini_cluster.cc:1428] Setting key 2a0826e8dc6a300775361cffbc1faea4
I20260924 16:14:20.185526 15596 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } attempt: 3
I20260924 16:14:20.186995 15672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:14:20 dist-test-slave-6wvm krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.125@KRBTEST.COM for kudu/127.1.253.124@KRBTEST.COM
Sep 24 16:14:20 dist-test-slave-6wvm krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.124@KRBTEST.COM
Sep 24 16:14:20 dist-test-slave-6wvm krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.124@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:20.199882 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.186006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34142)
0924 16:14:20.186193 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.186210 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:20.186319 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:20.187928 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:20.187931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:20.188143 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:20.188269 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:20.188276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.190264 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.190271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:20.190835 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:20.190845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.191719 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.191723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:20.191813 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.192372 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:20.192392 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:20.194055 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:20.197491 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:20.197498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:20.197501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:20.197733 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:20.198404 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:20.198407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:20.198409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:20.198503 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:20.199260 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:20.199264 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:20.199388 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 16:14:20.199508 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 16:14:20.199679 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 16:14:20.200835 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.189233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.1.253.126:34395)
0924 16:14:20.189365 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:20.189370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:20.189753 (+ 383us) server_negotiation.cc:407] Connection header received
0924 16:14:20.189906 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:20.189910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:20.189958 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:20.190002 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:20.190970 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.191676 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.192998 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.193449 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.193589 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.196949 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:20.196952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:20.196954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:20.196984 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:20.198747 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.199290 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.199294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.199295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.199397 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.199744 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.199747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.199749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.200017 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:20.200088 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:20.200554 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 16:14:20.200685 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:14:20.201296 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.185665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.1.253.124:38323)
0924 16:14:20.186112 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:20.186133 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:20.186179 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:14:20.187072 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:20.187098 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:20.187589 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:20.187981 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:20.189010 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.190062 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.191252 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.191676 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.192173 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.194997 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:20.195002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:20.195004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:20.195032 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:20.197718 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.198350 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.198354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.198355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.198442 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.198814 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.198817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.198819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.199126 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:20.200113 (+ 987us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:20.200192 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 16:14:20.200304 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":325,"server-negotiator.queue_time_us":290,"spinlock_wait_cycles":9600,"thread_start_us":128,"threads_started":1}
I20260924 16:14:20.201339 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.189183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:34395 (local address 127.0.0.1:48264)
0924 16:14:20.189602 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.189650 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:20.189771 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:20.190184 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:20.190190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:20.190680 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:20.190819 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:20.190826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.191811 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.191814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:20.192797 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:20.192811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.193646 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.193671 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:20.193789 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.194704 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:20.194721 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:20.196556 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:20.198877 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:20.198887 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:20.198894 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:20.199171 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:20.199534 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:20.199537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:20.199539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:20.199627 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:20.200167 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:20.200174 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:20.200291 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 16:14:20.200986 (+ 695us) client_negotiation.cc:240] Negotiation successful
0924 16:14:20.201179 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"spinlock_wait_cycles":2432,"thread_start_us":114,"threads_started":1}
I20260924 16:14:20.201309 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.186291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.1.253.124:38323)
0924 16:14:20.186606 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:20.186614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:20.186627 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:14:20.187072 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:20.187097 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:20.187589 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:20.187981 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:20.188525 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.190062 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.190962 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.191580 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.191756 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.194223 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:20.194232 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:20.194274 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:20.194313 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:20.197307 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.198075 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.198081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.198088 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.198214 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.198633 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.198636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.198638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.199122 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:20.199296 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:20.199707 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 16:14:20.200080 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":362,"server-negotiator.queue_time_us":243,"spinlock_wait_cycles":13440,"thread_start_us":137,"threads_started":1}
I20260924 16:14:20.203465 15672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:14:20.205940 15596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e: Bootstrap starting.
I20260924 16:14:20.208863 15596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:20.209745 15596 log.cc:824] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:20.211748 15596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e: No bootstrap required, opened a new log
Sep 24 16:14:20 dist-test-slave-6wvm krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
Sep 24 16:14:20 dist-test-slave-6wvm krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.124@KRBTEST.COM for kudu/127.1.253.125@KRBTEST.COM
I20260924 16:14:20.216243 15596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } }
I20260924 16:14:20.216538 15596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:20.216600 15596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6fa4b44a694d63811ac719ec983b4e, State: Initialized, Role: FOLLOWER
I20260924 16:14:20.216698 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.203920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.1.253.126:34395)
0924 16:14:20.204099 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:20.204103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:20.206493 (+ 2390us) server_negotiation.cc:407] Connection header received
0924 16:14:20.206576 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:20.206579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:20.206629 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:20.206708 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:20.207619 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.208327 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.209202 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.209540 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.209645 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.212515 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:20.212519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:20.212522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:20.212551 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:20.214621 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.215162 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.215167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.215169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.215268 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.215573 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.215577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.215579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.215920 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:20.216190 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:20.216367 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 16:14:20.216504 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:14:20.217198 15596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } }
I20260924 16:14:20.217361 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.204062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:43443 (local address 127.0.0.1:46046)
0924 16:14:20.204206 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.204223 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:20.204302 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:20.206998 (+ 2696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:20.207002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:20.207171 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:20.207288 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:20.207294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.208341 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.208345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:20.209184 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:20.209195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.209891 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.209894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:20.210031 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.211309 (+ 1278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:20.211326 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:20.212880 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:20.215405 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:20.215408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:20.215410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:20.215664 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:20.215993 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:20.215996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:20.215998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:20.216100 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:20.216788 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:20.216791 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:20.216919 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 16:14:20.217057 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 16:14:20.217200 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 16:14:20.217473 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.206638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.1.253.125:43443)
0924 16:14:20.206802 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:20.206806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:20.206817 (+ 11us) server_negotiation.cc:407] Connection header received
0924 16:14:20.206849 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:20.206850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:20.206886 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:20.206979 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:20.207408 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.208215 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.209303 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.209653 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.209746 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.213029 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:20.213033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:20.213036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:20.213063 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:20.215025 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.215780 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.215784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.215786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.215879 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.216219 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.216223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.216224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.216474 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:20.216555 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:20.217136 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 16:14:20.217274 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:14:20.218161 15672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:14:20.219177 15596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 16:14:20 dist-test-slave-6wvm krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.125@KRBTEST.COM
W20260924 16:14:20.225061 15690 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
Sep 24 16:14:20 dist-test-slave-6wvm krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266460, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.124@KRBTEST.COM for kudu/127.1.253.124@KRBTEST.COM
W20260924 16:14:20.225143 15690 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:14:20.223409 15528 master_runner.cc:186] Error getting master registration for 127.1.253.126:34395: OK, instance_id { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" instance_seqno: 1790266459700439 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:14:20.226711 15528 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:14:20.226953 15679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } } }
I20260924 16:14:20.227078 15679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:14:20.229557 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.218789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34152)
0924 16:14:20.218920 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.218934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:20.219005 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:20.219386 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:20.219389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:20.220596 (+ 1207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:20.220714 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:20.220725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.221858 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.221861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:20.222684 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:20.222697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.223392 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.223398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:20.223501 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.223985 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:20.224005 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:20.225446 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:20.227770 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:20.227773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:20.227775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:20.228024 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:20.228367 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:20.228371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:20.228374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:20.228466 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:20.228969 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:20.228973 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:20.229082 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 16:14:20.229192 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 16:14:20.229366 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 16:14:20.229682 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.219020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.1.253.124:38323)
0924 16:14:20.219151 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:20.219156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:20.219171 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:14:20.219213 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:20.219217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:20.219258 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:20.219347 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:20.220895 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.221704 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.222842 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.223252 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.223717 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.225575 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:20.225579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:20.225581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:20.225604 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:20.227647 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.228150 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.228154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.228155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.228254 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.228566 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.228569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.228571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.228874 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:20.228983 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:20.229269 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 16:14:20.229444 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":44}
I20260924 16:14:20.230634 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.218389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46056 (local address 127.1.253.125:43443)
0924 16:14:20.218494 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:20.218498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:20.218511 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:14:20.218546 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:20.218549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:20.218591 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:20.218672 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:20.219480 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.220263 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.221452 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.221786 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.221879 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.226662 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:20.226672 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:20.226676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:20.226709 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:20.228457 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.228891 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.228895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.228897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.228980 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.229299 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.229305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.229309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.229525 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:20.229752 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:20.230373 (+ 621us) server_negotiation.cc:299] Negotiation successful
0924 16:14:20.230490 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:14:20.233137 15672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157: Bootstrap starting.
I20260924 16:14:20.234426 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:14:20.235798 15672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:20.236608 15672 log.cc:824] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:20.241492 15672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157: No bootstrap required, opened a new log
I20260924 16:14:20.246565 15672 raft_consensus.cc:348] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } }
I20260924 16:14:20.247733 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.237700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.1.253.125:43443)
0924 16:14:20.237809 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:20.237813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:20.238175 (+ 362us) server_negotiation.cc:407] Connection header received
0924 16:14:20.238394 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:20.238399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:20.238455 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:20.238513 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:20.239850 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.240581 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.241634 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.241935 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.242018 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.243935 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:20.243938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:20.243941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:20.243970 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:20.245724 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.246216 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.246220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.246222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.246367 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.246689 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.246692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.246694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.246931 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:20.247042 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:20.247439 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 16:14:20.247567 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 16:14:20.248242 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.237252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.1.253.124:38323)
0924 16:14:20.237378 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:20.237383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:20.238123 (+ 740us) server_negotiation.cc:407] Connection header received
0924 16:14:20.238348 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:20.238351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:20.238398 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:20.238449 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:20.239330 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.240065 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.241274 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.241641 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.241717 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.243931 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:20.243935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:20.243938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:20.243965 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:20.246400 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.246890 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.246893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.246895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.246978 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.247268 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.247271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.247273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.247500 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:20.247625 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:20.247969 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 16:14:20.248091 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 16:14:20.250567 15672 raft_consensus.cc:374] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:20.251017 15672 raft_consensus.cc:729] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44943d2273a4634afe254561539f157, State: Initialized, Role: FOLLOWER
I20260924 16:14:20.251868 15672 consensus_queue.cc:260] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } }
I20260924 16:14:20.252418 15700 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } } }
I20260924 16:14:20.252604 15700 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:14:20.253368 15672 sys_catalog.cc:564] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:14:20.253973 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.237519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.1.253.126:34395)
0924 16:14:20.237750 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:20.237754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:20.237770 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:14:20.237827 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:20.237831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:20.237890 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:20.237973 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:20.239084 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.239842 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.248288 (+ 8446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.248624 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.248716 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.250339 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:20.250343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:20.250350 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:20.250390 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:20.252075 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.252578 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.252582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.252584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.252676 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.252990 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.252994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.252996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.253208 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:20.253307 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:20.253681 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 16:14:20.253812 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260924 16:14:20.256381 15679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:14:20.256549 15679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } }
I20260924 16:14:20.257579 15679 leader_election.cc:288] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a5d707d6a3453f868959cb8815b328 (127.1.253.126:34395), f44943d2273a4634afe254561539f157 (127.1.253.124:38323)
I20260924 16:14:20.258308 15646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c6fa4b44a694d63811ac719ec983b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44943d2273a4634afe254561539f157" is_pre_election: true
I20260924 16:14:20.258605 15646 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6fa4b44a694d63811ac719ec983b4e in term 0.
I20260924 16:14:20.259053 15544 leader_election.cc:302] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6fa4b44a694d63811ac719ec983b4e, f44943d2273a4634afe254561539f157; no voters:
I20260924 16:14:20.259366 15679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:14:20.259471 15679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:14:20.259514 15679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:20.260845 15679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } }
I20260924 16:14:20.257618 15603 master_runner.cc:186] Error getting master registration for 127.1.253.126:34395: OK, instance_id { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" instance_seqno: 1790266459700439 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:14:20.261070 15603 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:14:20.261225 15679 leader_election.cc:288] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [CANDIDATE]: Term 1 election: Requested vote from peers 52a5d707d6a3453f868959cb8815b328 (127.1.253.126:34395), f44943d2273a4634afe254561539f157 (127.1.253.124:38323)
I20260924 16:14:20.261615 15646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c6fa4b44a694d63811ac719ec983b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44943d2273a4634afe254561539f157"
I20260924 16:14:20.261739 15646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:20.263145 15646 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6fa4b44a694d63811ac719ec983b4e in term 1.
I20260924 16:14:20.263453 15544 leader_election.cc:302] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6fa4b44a694d63811ac719ec983b4e, f44943d2273a4634afe254561539f157; no voters:
I20260924 16:14:20.263662 15679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:14:20.263882 15679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [term 1 LEADER]: Becoming Leader. State: Replica: 3c6fa4b44a694d63811ac719ec983b4e, State: Running, Role: LEADER
I20260924 16:14:20.264237 15679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } }
W20260924 16:14:20.264917 15711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:14:20.264987 15711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:14:20.266852 15713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c6fa4b44a694d63811ac719ec983b4e. Latest consensus state: current_term: 1 leader_uuid: "3c6fa4b44a694d63811ac719ec983b4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } } }
I20260924 16:14:20.267182 15713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:20.273974 15715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:14:20.275590 15715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:14:20.281884 15646 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [term 1 FOLLOWER]: Refusing update from remote peer 3c6fa4b44a694d63811ac719ec983b4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:14:20.282819 15713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:14:20.288908 15700 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c6fa4b44a694d63811ac719ec983b4e. Latest consensus state: current_term: 1 leader_uuid: "3c6fa4b44a694d63811ac719ec983b4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } } }
I20260924 16:14:20.289041 15700 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:14:20.292284 15679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c6fa4b44a694d63811ac719ec983b4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } } }
I20260924 16:14:20.292438 15679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:20.292547 15716 mvcc.cc:205] Tried to move back new op lower bound from 7332931421306425344 to 7332931421245440000. Current Snapshot: MvccSnapshot[applied={T|T < 7332931421306425344}]
I20260924 16:14:20.292781 15700 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c6fa4b44a694d63811ac719ec983b4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } } }
I20260924 16:14:20.292908 15700 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:14:20.294520 15715 catalog_manager.cc:1347] Generated new cluster ID: e4c3c4fc0cd04ac2841b04328bff3a0b
I20260924 16:14:20.294595 15715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:14:20.294682 15717 mvcc.cc:205] Tried to move back new op lower bound from 7332931421306425344 to 7332931421245440000. Current Snapshot: MvccSnapshot[applied={T|T < 7332931421306425344}]
I20260924 16:14:20.328361 15715 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:14:20.330385 15715 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:14:20.339039 15715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e: Generated new TSK 0
I20260924 16:14:20.339865 15715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:14:20.341151 15521 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } attempt: 4
Sep 24 16:14:20 dist-test-slave-6wvm krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266459, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.124@KRBTEST.COM
I20260924 16:14:20.353543 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.341854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.1.253.124:38323)
0924 16:14:20.341969 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:20.341973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:20.341986 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:14:20.342025 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:20.342028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:20.342074 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:20.342152 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:20.342785 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.343571 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.344668 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.345019 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.345142 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.347630 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:20.347634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:20.347637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:20.347666 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:20.349534 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.350063 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.350067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.350068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.350158 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:20.350516 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:20.350519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:20.350521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:20.350740 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:20.350829 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:20.353230 (+ 2401us) server_negotiation.cc:299] Negotiation successful
0924 16:14:20.353378 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 16:14:20.354097 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:20.341609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:38323 (local address 127.0.0.1:34176)
0924 16:14:20.341762 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:20.341781 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:20.341878 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:20.342301 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:20.342305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:20.342529 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:20.342638 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:20.342645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.343705 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.343708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:20.344540 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:20.344555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:20.345164 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:20.345168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:20.345255 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:20.345850 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:20.345867 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:20.347482 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:20.349662 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:20.349666 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:20.349668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:20.349955 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:20.350298 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:20.350301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:20.350312 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:20.350407 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:20.350877 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:20.350881 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:20.350985 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 16:14:20.351078 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:14:20.351223 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 16:14:20.360110 15521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328: Bootstrap starting.
I20260924 16:14:20.364209 15521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:20.365262 15521 log.cc:824] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:20.367852 15521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328: No bootstrap required, opened a new log
I20260924 16:14:20.372329 15521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } }
I20260924 16:14:20.372651 15521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:20.372715 15521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a5d707d6a3453f868959cb8815b328, State: Initialized, Role: FOLLOWER
I20260924 16:14:20.373330 15521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } }
I20260924 16:14:20.374132 15726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } } }
I20260924 16:14:20.374536 15726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:14:20.374945 15521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:14:20.375185 15493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:20.377386 15493 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [term 1 FOLLOWER]: Refusing update from remote peer 3c6fa4b44a694d63811ac719ec983b4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:14:20.378520 15679 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:14:20.378590 15494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c6fa4b44a694d63811ac719ec983b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a5d707d6a3453f868959cb8815b328"
I20260924 16:14:20.380036 15494 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6fa4b44a694d63811ac719ec983b4e in term 1.
I20260924 16:14:20.380344 15495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c6fa4b44a694d63811ac719ec983b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a5d707d6a3453f868959cb8815b328" is_pre_election: true
I20260924 16:14:20.380487 15495 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3c6fa4b44a694d63811ac719ec983b4e in term 1. Re-sending same reply.
I20260924 16:14:20.386654 15726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c6fa4b44a694d63811ac719ec983b4e. Latest consensus state: current_term: 1 leader_uuid: "3c6fa4b44a694d63811ac719ec983b4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } } }
I20260924 16:14:20.386804 15726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:14:20.386902 15726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c6fa4b44a694d63811ac719ec983b4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } } }
I20260924 16:14:20.386965 15726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:14:20.388622 15737 mvcc.cc:205] Tried to move back new op lower bound from 7332931421536862208 to 7332931421245440000. Current Snapshot: MvccSnapshot[applied={T|T < 7332931421306425344}]
W20260924 16:14:20.393994 15741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:14:20.396859 15741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:14:20.774895 15679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c6fa4b44a694d63811ac719ec983b4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a5d707d6a3453f868959cb8815b328" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34395 } } peers { permanent_uuid: "3c6fa4b44a694d63811ac719ec983b4e" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 43443 } } peers { permanent_uuid: "f44943d2273a4634afe254561539f157" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 38323 } } }
I20260924 16:14:20.775180 15679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6fa4b44a694d63811ac719ec983b4e [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:21.266558 15711 catalog_manager.cc:1259] Loaded cluster ID: e4c3c4fc0cd04ac2841b04328bff3a0b
I20260924 16:14:21.266645 15711 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157: loading cluster ID for follower catalog manager: success
I20260924 16:14:21.268366 15711 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157: acquiring CA information for follower catalog manager: success
I20260924 16:14:21.269585 15711 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f44943d2273a4634afe254561539f157: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:14:21.398418 15741 catalog_manager.cc:1259] Loaded cluster ID: e4c3c4fc0cd04ac2841b04328bff3a0b
I20260924 16:14:21.398517 15741 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328: loading cluster ID for follower catalog manager: success
I20260924 16:14:21.400166 15741 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328: acquiring CA information for follower catalog manager: success
I20260924 16:14:21.401348 15741 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 52a5d707d6a3453f868959cb8815b328: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:14:21.422602 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 15453
I20260924 16:14:21.430809 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 15528
I20260924 16:14:21.437618 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 15603
2026-09-24T16:14:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3110 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:14:21 dist-test-slave-6wvm krb5kdc[15748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:14:21 dist-test-slave-6wvm krb5kdc[15748](info): set up 2 sockets
Sep 24 16:14:21 dist-test-slave-6wvm krb5kdc[15748](info): commencing operation
krb5kdc: starting...
W20260924 16:14:22.494441 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:14:22 dist-test-slave-6wvm krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:14:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:14:22.626471 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:34913
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:34913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:22.751674 15761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:22.751988 15761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:22.752034 15761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:22.756255 15761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:14:22.756335 15761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:22.756356 15761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:22.756376 15761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:14:22.756395 15761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:14:22.761860 15761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:34913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:34913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:22.763011 15761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:22.764183 15761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:22.773160 15769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:22.773205 15761 server_base.cc:1034] running on GCE node
W20260924 16:14:22.773123 15767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:22.773149 15766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:22.774138 15761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:22.775305 15761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:22.776467 15761 hybrid_clock.cc:648] HybridClock initialized: now 1790266462776452 us; error 25 us; skew 500 ppm
Sep 24 16:14:22 dist-test-slave-6wvm krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266462, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:22.779398 15761 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:14:22.780339 15761 webserver.cc:460] Webserver started at http://127.1.253.126:37289/ using document root <none> and password file <none>
I20260924 16:14:22.780938 15761 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:22.781045 15761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:22.781293 15761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:14:22.783507 15761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "aed4666c04c84181a84106f3b29f846b"
format_stamp: "Formatted at 2026-09-24 16:14:22 on dist-test-slave-6wvm"
server_key: "1e1bbc8382b1084f02a6beda0956da54"
server_key_iv: "005bf4780fa0d17818b864fa29c0f376"
server_key_version: "encryptionkey@0"
I20260924 16:14:22.784126 15761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "aed4666c04c84181a84106f3b29f846b"
format_stamp: "Formatted at 2026-09-24 16:14:22 on dist-test-slave-6wvm"
server_key: "1e1bbc8382b1084f02a6beda0956da54"
server_key_iv: "005bf4780fa0d17818b864fa29c0f376"
server_key_version: "encryptionkey@0"
I20260924 16:14:22.788125 15761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:14:22.790745 15776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:22.792105 15761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:14:22.792268 15761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "aed4666c04c84181a84106f3b29f846b"
format_stamp: "Formatted at 2026-09-24 16:14:22 on dist-test-slave-6wvm"
server_key: "1e1bbc8382b1084f02a6beda0956da54"
server_key_iv: "005bf4780fa0d17818b864fa29c0f376"
server_key_version: "encryptionkey@0"
I20260924 16:14:22.792394 15761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:22.809089 15761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:22.812770 15761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:22.813010 15761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:22.820436 15761 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:34913
I20260924 16:14:22.820494 15828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:34913 every 8 connection(s)
I20260924 16:14:22.821552 15761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:14:22.824460 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 15761
I20260924 16:14:22.824584 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:14:22.824646 15829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:22.824916 2037 external_mini_cluster.cc:1428] Setting key 343196a9a89b2265288c94f0237cf07e
I20260924 16:14:22.832809 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b: Bootstrap starting.
Sep 24 16:14:22 dist-test-slave-6wvm krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:22.835464 15829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:22.836433 15829 log.cc:824] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:22.838967 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b: No bootstrap required, opened a new log
I20260924 16:14:22.841065 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:22.826379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37922 (local address 127.1.253.126:34913)
0924 16:14:22.826838 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:22.826854 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:22.826894 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:14:22.827788 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:22.827811 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:22.828192 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:22.828509 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:22.829560 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:22.830687 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:22.832001 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:22.832693 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:22.832877 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:22.835512 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:22.835521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:22.835540 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:22.835575 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:22.837550 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:22.837992 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:22.837998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:22.838005 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:22.838147 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:22.838475 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:22.838478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:22.838479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:22.839014 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:22.839210 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:22.839486 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 16:14:22.839809 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":149,"threads_started":1}
I20260924 16:14:22.842846 15829 raft_consensus.cc:348] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed4666c04c84181a84106f3b29f846b" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34913 } }
I20260924 16:14:22.843168 15829 raft_consensus.cc:374] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:22.843240 15829 raft_consensus.cc:729] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed4666c04c84181a84106f3b29f846b, State: Initialized, Role: FOLLOWER
I20260924 16:14:22.843705 15829 consensus_queue.cc:260] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed4666c04c84181a84106f3b29f846b" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34913 } }
I20260924 16:14:22.843878 15829 raft_consensus.cc:388] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:14:22.843950 15829 raft_consensus.cc:482] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:14:22.844048 15829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:22.844348 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:14:22.845271 15829 raft_consensus.cc:504] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed4666c04c84181a84106f3b29f846b" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34913 } }
I20260924 16:14:22.845680 15829 leader_election.cc:302] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aed4666c04c84181a84106f3b29f846b; no voters:
I20260924 16:14:22.846050 15829 leader_election.cc:288] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:14:22.846210 15834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:14:22.846490 15834 raft_consensus.cc:686] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [term 1 LEADER]: Becoming Leader. State: Replica: aed4666c04c84181a84106f3b29f846b, State: Running, Role: LEADER
I20260924 16:14:22.846844 15834 consensus_queue.cc:237] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed4666c04c84181a84106f3b29f846b" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34913 } }
I20260924 16:14:22.847348 15829 sys_catalog.cc:564] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:14:22.848598 15836 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aed4666c04c84181a84106f3b29f846b. Latest consensus state: current_term: 1 leader_uuid: "aed4666c04c84181a84106f3b29f846b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed4666c04c84181a84106f3b29f846b" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34913 } } }
I20260924 16:14:22.848723 15836 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:22.849071 15835 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aed4666c04c84181a84106f3b29f846b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed4666c04c84181a84106f3b29f846b" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 34913 } } }
I20260924 16:14:22.849161 15835 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:22.849248 15843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:14:22.852617 15843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:14:22.858564 15843 catalog_manager.cc:1347] Generated new cluster ID: 3e9da5c13455407ea8c149f705c31c4a
I20260924 16:14:22.858655 15843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:14:22.875552 15843 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:14:22.876611 15843 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:14:22.883483 15843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aed4666c04c84181a84106f3b29f846b: Generated new TSK 0
I20260924 16:14:22.884699 15843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:14:22.901125 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 15761
2026-09-24T16:14:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1461 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:14:22 dist-test-slave-6wvm krb5kdc[15856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:14:22 dist-test-slave-6wvm krb5kdc[15856](info): set up 2 sockets
Sep 24 16:14:22 dist-test-slave-6wvm krb5kdc[15856](info): commencing operation
krb5kdc: starting...
W20260924 16:14:23.954731 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:14:24 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:14:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:14:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:14:24.083722 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:45771
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:45771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:24.207971 15869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:24.208320 15869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:24.208417 15869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:24.213294 15869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:14:24.213373 15869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:24.213415 15869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:24.213456 15869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:14:24.213497 15869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:14:24.221598 15869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:45771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:45771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:24.223050 15869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:24.224387 15869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:14:24.233808 15869 server_base.cc:1034] running on GCE node
W20260924 16:14:24.233683 15877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:24.233671 15875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:24.233671 15874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:24.234783 15869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:24.235972 15869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:24.237172 15869 hybrid_clock.cc:648] HybridClock initialized: now 1790266464237146 us; error 37 us; skew 500 ppm
Sep 24 16:14:24 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:24.240087 15869 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:14:24.241098 15869 webserver.cc:460] Webserver started at http://127.1.253.126:41869/ using document root <none> and password file <none>
I20260924 16:14:24.241727 15869 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:24.241830 15869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:24.242080 15869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:14:24.244271 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "76876cc5fc2d4c7fb62f865db014fe49"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "2f01d4ff54eb8b8087459ebb76d5e933"
server_key_iv: "d5dc6962a82f5e34b844ede353e3a743"
server_key_version: "encryptionkey@0"
I20260924 16:14:24.244863 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "76876cc5fc2d4c7fb62f865db014fe49"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "2f01d4ff54eb8b8087459ebb76d5e933"
server_key_iv: "d5dc6962a82f5e34b844ede353e3a743"
server_key_version: "encryptionkey@0"
I20260924 16:14:24.248682 15869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 16:14:24.251221 15884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:24.252512 15869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:14:24.252660 15869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "76876cc5fc2d4c7fb62f865db014fe49"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "2f01d4ff54eb8b8087459ebb76d5e933"
server_key_iv: "d5dc6962a82f5e34b844ede353e3a743"
server_key_version: "encryptionkey@0"
I20260924 16:14:24.252786 15869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:24.284507 15869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:24.287617 15869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:24.287846 15869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:24.295220 15869 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:45771
I20260924 16:14:24.295518 15936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:45771 every 8 connection(s)
I20260924 16:14:24.296347 15869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:14:24.299414 15937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:24.304005 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 15869
I20260924 16:14:24.304139 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:14:24.304461 2037 external_mini_cluster.cc:1428] Setting key 052bfed57ec1a1aaad6fb4915cffc319
I20260924 16:14:24.305685 15937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Bootstrap starting.
I20260924 16:14:24.308096 15937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:24.308871 15937 log.cc:824] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:24.310946 15937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: No bootstrap required, opened a new log
Sep 24 16:14:24 dist-test-slave-6wvm krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:24.315470 15937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } }
I20260924 16:14:24.315807 15937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:24.315853 15937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76876cc5fc2d4c7fb62f865db014fe49, State: Initialized, Role: FOLLOWER
I20260924 16:14:24.316344 15937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } }
I20260924 16:14:24.316465 15937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:14:24.316560 15937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:14:24.316681 15937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:24.317925 15937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } }
I20260924 16:14:24.318346 15937 leader_election.cc:302] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76876cc5fc2d4c7fb62f865db014fe49; no voters:
I20260924 16:14:24.318738 15937 leader_election.cc:288] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:14:24.318876 15942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:14:24.319280 15942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 1 LEADER]: Becoming Leader. State: Replica: 76876cc5fc2d4c7fb62f865db014fe49, State: Running, Role: LEADER
I20260924 16:14:24.319612 15942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } }
I20260924 16:14:24.320190 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:24.306028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.1.253.126:45771)
0924 16:14:24.306530 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:24.306546 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:24.306583 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:14:24.307220 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:24.307237 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:24.307611 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:24.307962 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:24.308817 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:24.310086 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:24.311032 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:24.311545 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:24.312098 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:24.314646 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:24.314652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:24.314666 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:24.314695 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:24.316766 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:24.317260 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:24.317265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:24.317269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:24.317393 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:24.317720 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:24.317722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:24.317723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:24.318187 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:24.318429 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:24.318751 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 16:14:24.319155 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":186,"threads_started":1}
I20260924 16:14:24.320405 15937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:14:24.320883 15942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76876cc5fc2d4c7fb62f865db014fe49. Latest consensus state: current_term: 1 leader_uuid: "76876cc5fc2d4c7fb62f865db014fe49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } } }
I20260924 16:14:24.320984 15942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:24.321406 15946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:14:24.323340 15943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76876cc5fc2d4c7fb62f865db014fe49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } } }
I20260924 16:14:24.323508 15943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:24.324667 15946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:14:24.330600 15946 catalog_manager.cc:1347] Generated new cluster ID: 301e9ca6e8634dffba066a28ae2305ef
I20260924 16:14:24.330675 15946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:14:24.345842 15946 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:14:24.347257 15946 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:14:24.358596 15946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Generated new TSK 0
I20260924 16:14:24.359405 15946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:14:24.394645 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45771
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:24.525334 15963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:24.525647 15963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:24.525694 15963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:24.529834 15963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:24.529902 15963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:24.529963 15963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:14:24.535287 15963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:24.536300 15963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:24.537324 15963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:24.548491 15971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:24.548520 15968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:24.548617 15963 server_base.cc:1034] running on GCE node
W20260924 16:14:24.548785 15969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:24.549259 15963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:24.549950 15963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:24.551119 15963 hybrid_clock.cc:648] HybridClock initialized: now 1790266464551071 us; error 57 us; skew 500 ppm
Sep 24 16:14:24 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:24.553926 15963 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:14:24.554878 15963 webserver.cc:460] Webserver started at http://127.1.253.65:41429/ using document root <none> and password file <none>
I20260924 16:14:24.555464 15963 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:24.555558 15963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:24.555796 15963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:14:24.557885 15963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "2353fc3ef96145278fc62b01dde95288"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "d3922a3f99782138e901eebc45376e0c"
server_key_iv: "af11e315ca356a5a7160c0507cacd9d3"
server_key_version: "encryptionkey@0"
I20260924 16:14:24.558506 15963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "2353fc3ef96145278fc62b01dde95288"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "d3922a3f99782138e901eebc45376e0c"
server_key_iv: "af11e315ca356a5a7160c0507cacd9d3"
server_key_version: "encryptionkey@0"
I20260924 16:14:24.562404 15963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:14:24.564891 15978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:24.566092 15963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:14:24.566286 15963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "2353fc3ef96145278fc62b01dde95288"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "d3922a3f99782138e901eebc45376e0c"
server_key_iv: "af11e315ca356a5a7160c0507cacd9d3"
server_key_version: "encryptionkey@0"
I20260924 16:14:24.566416 15963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:24.583479 15963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:24.586561 15963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:24.586774 15963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:24.587414 15963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:24.588299 15963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:24.588373 15963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:24.588451 15963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:24.588495 15963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:24.597949 15963 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:38779
I20260924 16:14:24.598012 16092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:38779 every 8 connection(s)
I20260924 16:14:24.598981 15963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:14:24.603107 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 15963
I20260924 16:14:24.603266 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:14:24.603562 2037 external_mini_cluster.cc:1428] Setting key f9b80015b3520b12c32bc4966f1d4426
Sep 24 16:14:24 dist-test-slave-6wvm krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:14:24.614531 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:24.600993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:58513 (local address 127.1.253.126:45771)
0924 16:14:24.601137 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:24.601141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:24.602126 (+ 985us) server_negotiation.cc:407] Connection header received
0924 16:14:24.603222 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:24.603229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:24.603316 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:24.603419 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:24.604972 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:24.605749 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:24.607046 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:24.607281 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:24.607400 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:24.610259 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:24.610263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:24.610266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:24.610304 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:24.611840 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:24.612420 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:24.612424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:24.612425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:24.612532 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:24.612897 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:24.612903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:24.612906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:24.613184 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:24.613314 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:24.614161 (+ 847us) server_negotiation.cc:299] Negotiation successful
0924 16:14:24.614355 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:14:24.615983 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:24.601296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:58513)
0924 16:14:24.601987 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:24.602031 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:24.602939 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:24.603620 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:24.603627 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:24.604131 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:24.604809 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:24.604830 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:24.605890 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:24.605893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:24.606900 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:24.606910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:24.607441 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:24.607446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:24.607629 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:24.608259 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:24.608285 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:24.610004 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:24.611969 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:24.611975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:24.611991 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:24.612293 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:24.612638 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:24.612641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:24.612642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:24.612781 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:24.613364 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:24.613373 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:24.613761 (+ 388us) client_negotiation.cc:769] Sending connection context
0924 16:14:24.614734 (+ 973us) client_negotiation.cc:240] Negotiation successful
0924 16:14:24.615086 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
I20260924 16:14:24.617801 16093 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45771
I20260924 16:14:24.618083 16093 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:24.618724 16093 heartbeater.cc:511] Master 127.1.253.126:45771 requested a full tablet report, sending...
I20260924 16:14:24.620805 15901 ts_manager.cc:194] Registered new tserver with Master: 2353fc3ef96145278fc62b01dde95288 (127.1.253.65:38779)
I20260924 16:14:24.622113 15901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:58513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:14:24.637455 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45771
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:24.754874 16098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:24.755174 16098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:24.755219 16098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:24.759472 16098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:24.759539 16098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:24.759622 16098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:14:24.765198 16098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:24.766454 16098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:24.767544 16098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:24.777899 16106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:24.777899 16104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:24.777902 16103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:24.778187 16098 server_base.cc:1034] running on GCE node
I20260924 16:14:24.778692 16098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:24.779325 16098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:24.780490 16098 hybrid_clock.cc:648] HybridClock initialized: now 1790266464780473 us; error 28 us; skew 500 ppm
Sep 24 16:14:24 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:24.793861 16098 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:14:24.795037 16098 webserver.cc:460] Webserver started at http://127.1.253.66:41847/ using document root <none> and password file <none>
I20260924 16:14:24.795684 16098 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:24.795787 16098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:24.796053 16098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:14:24.798363 16098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "bcbc0826a058433092d482b61f0d4b63"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "c7edb68bf1c0010bcb9c8a942dc11073"
server_key_iv: "69d56ec57fced7665a39b654c3c2183a"
server_key_version: "encryptionkey@0"
I20260924 16:14:24.798992 16098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "bcbc0826a058433092d482b61f0d4b63"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "c7edb68bf1c0010bcb9c8a942dc11073"
server_key_iv: "69d56ec57fced7665a39b654c3c2183a"
server_key_version: "encryptionkey@0"
I20260924 16:14:24.803244 16098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:14:24.806088 16113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:24.807531 16098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:14:24.807682 16098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "bcbc0826a058433092d482b61f0d4b63"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "c7edb68bf1c0010bcb9c8a942dc11073"
server_key_iv: "69d56ec57fced7665a39b654c3c2183a"
server_key_version: "encryptionkey@0"
I20260924 16:14:24.807806 16098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:24.823977 16098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:24.827664 16098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:24.827883 16098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:24.828522 16098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:24.829409 16098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:24.829479 16098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:24.829555 16098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:24.829605 16098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:24.838955 16098 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:42023
I20260924 16:14:24.838975 16228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:42023 every 8 connection(s)
I20260924 16:14:24.839950 16098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:14:24.845098 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 16098
I20260924 16:14:24.845238 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:14:24.845573 2037 external_mini_cluster.cc:1428] Setting key edc79ca1dbea2b21e1b6a0be07eb3a59
Sep 24 16:14:24 dist-test-slave-6wvm krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266464, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:24.855572 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:24.841960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:54605 (local address 127.1.253.126:45771)
0924 16:14:24.842114 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:24.842118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:24.843307 (+ 1189us) server_negotiation.cc:407] Connection header received
0924 16:14:24.844169 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:24.844172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:24.844220 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:24.844298 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:24.845852 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:24.846533 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:24.847951 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:24.848347 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:24.848480 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:24.851388 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:24.851391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:24.851393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:24.851417 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:24.852952 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:24.853693 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:24.853696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:24.853697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:24.853775 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:24.854226 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:24.854229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:24.854230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:24.854494 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:24.854591 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:24.855327 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 16:14:24.855439 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:14:24.857074 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:24.842452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:54605)
0924 16:14:24.843145 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:24.843189 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:24.843973 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:24.844468 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:24.844475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:24.844921 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:24.845699 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:24.845721 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:24.846661 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:24.846669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:24.847817 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:24.847828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:24.848512 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:24.848514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:24.848658 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:24.849336 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:24.849353 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:24.851202 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:24.853103 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:24.853113 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:24.853136 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:24.853588 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:24.853877 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:24.853884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:24.853887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:24.854117 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:24.854627 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:24.854641 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:24.855032 (+ 391us) client_negotiation.cc:769] Sending connection context
0924 16:14:24.855282 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 16:14:24.855704 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":120,"threads_started":1}
I20260924 16:14:24.858569 16229 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45771
I20260924 16:14:24.858881 16229 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:24.859546 16229 heartbeater.cc:511] Master 127.1.253.126:45771 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
I20260924 16:14:24.860790 15901 ts_manager.cc:194] Registered new tserver with Master: bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023)
I20260924 16:14:24.861415 15901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:54605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:14:24.877513 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45771
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:25.000236 16234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:25.000593 16234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:25.000692 16234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:25.004908 16234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:25.004981 16234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:25.005110 16234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:14:25.010643 16234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:45771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:25.011901 16234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:25.013056 16234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:25.024286 16239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:25.024309 16240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:25.024428 16234 server_base.cc:1034] running on GCE node
W20260924 16:14:25.024294 16242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:25.025063 16234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:25.025712 16234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:25.026885 16234 hybrid_clock.cc:648] HybridClock initialized: now 1790266465026864 us; error 32 us; skew 500 ppm
Sep 24 16:14:25 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:25.029690 16234 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:14:25.030787 16234 webserver.cc:460] Webserver started at http://127.1.253.67:41837/ using document root <none> and password file <none>
I20260924 16:14:25.031383 16234 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:25.031445 16234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:25.031618 16234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:14:25.033720 16234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "e58776ae6a124a0da6a9c3541adba1c8"
format_stamp: "Formatted at 2026-09-24 16:14:25 on dist-test-slave-6wvm"
server_key: "982841ad1648d7fd12ae7c79fb2f49de"
server_key_iv: "8802374937963b8def61c05ab136dd8b"
server_key_version: "encryptionkey@0"
I20260924 16:14:25.034307 16234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "e58776ae6a124a0da6a9c3541adba1c8"
format_stamp: "Formatted at 2026-09-24 16:14:25 on dist-test-slave-6wvm"
server_key: "982841ad1648d7fd12ae7c79fb2f49de"
server_key_iv: "8802374937963b8def61c05ab136dd8b"
server_key_version: "encryptionkey@0"
I20260924 16:14:25.038120 16234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:14:25.040661 16249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:25.042027 16234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:14:25.042189 16234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "e58776ae6a124a0da6a9c3541adba1c8"
format_stamp: "Formatted at 2026-09-24 16:14:25 on dist-test-slave-6wvm"
server_key: "982841ad1648d7fd12ae7c79fb2f49de"
server_key_iv: "8802374937963b8def61c05ab136dd8b"
server_key_version: "encryptionkey@0"
I20260924 16:14:25.042372 16234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:25.080250 16234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:25.083863 16234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:25.084096 16234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:25.084751 16234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:25.085666 16234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:25.085742 16234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:25.085822 16234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:25.085879 16234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:25.095772 16234 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:36577
I20260924 16:14:25.095791 16363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:36577 every 8 connection(s)
I20260924 16:14:25.096755 16234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:14:25.105636 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 16234
I20260924 16:14:25.105778 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:14:25.106140 2037 external_mini_cluster.cc:1428] Setting key b2026b873c62fdd738845653d10563f4
Sep 24 16:14:25 dist-test-slave-6wvm krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:25.110984 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.098715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:41931 (local address 127.1.253.126:45771)
0924 16:14:25.098859 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:25.098863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:25.099892 (+ 1029us) server_negotiation.cc:407] Connection header received
0924 16:14:25.100914 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:25.100918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:25.100984 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:25.101100 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:25.102365 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.102904 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.104074 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.104304 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.104421 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.107417 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:25.107421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:25.107424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:25.107473 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:25.109076 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:25.109552 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:25.109555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:25.109557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:25.109631 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:25.109949 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:25.109951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:25.109952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:25.110129 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:25.110203 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:25.110737 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 16:14:25.110842 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:14:25.111932 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.099065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:41931)
0924 16:14:25.099735 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:25.099778 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:25.100687 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:25.101293 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:25.101300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:25.101669 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:25.102170 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.102184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.103046 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.103049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.103926 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.103936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.104468 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.104472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.104649 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.105345 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:25.105370 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:25.107231 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:25.109184 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:25.109189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:25.109201 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:25.109463 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:25.109711 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:25.109713 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:25.109715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:25.109850 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:25.110217 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:25.110222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:25.110522 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 16:14:25.110688 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 16:14:25.110951 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":113,"threads_started":1}
I20260924 16:14:25.113130 16364 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45771
I20260924 16:14:25.113404 16364 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:25.113948 16364 heartbeater.cc:511] Master 127.1.253.126:45771 requested a full tablet report, sending...
I20260924 16:14:25.115103 15901 ts_manager.cc:194] Registered new tserver with Master: e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577)
I20260924 16:14:25.115633 15901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:41931
I20260924 16:14:25.120410 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:14:25.129676 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.122621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.1.253.126:45771)
0924 16:14:25.122744 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:25.122748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:25.122924 (+ 176us) server_negotiation.cc:407] Connection header received
0924 16:14:25.123023 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:25.123026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:25.123064 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:25.123131 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:25.123781 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.124285 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.125105 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.125318 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.125403 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.126902 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:25.126905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:25.126908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:25.126929 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:25.128230 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:25.128620 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:25.128623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:25.128624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:25.128699 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:25.128937 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:25.128939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:25.128940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:25.129154 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:25.129260 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:25.129465 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 16:14:25.129558 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:14:25.132771 15901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:14:25.135811 15901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:14:25.150859 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.144547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:38779 (local address 127.0.0.1:59400)
0924 16:14:25.145590 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:25.145606 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:25.145700 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:25.146495 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:25.146499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:25.146519 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:25.146636 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.146646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.148208 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.148211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.149199 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.149209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.150254 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.150259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.150367 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.150404 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:14:25.150524 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 16:14:25.150698 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":28,"thread_start_us":118,"threads_started":1}
I20260924 16:14:25.150883 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.145208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.1.253.65:38779)
0924 16:14:25.145996 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:25.146005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:25.146033 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:14:25.146117 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:25.146123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:25.146349 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:25.146509 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:25.146776 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.148066 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.149346 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.150090 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.150258 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.150429 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:25.150575 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 16:14:25.150684 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":80,"threads_started":1}
I20260924 16:14:25.150906 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.144311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:36577 (local address 127.0.0.1:56398)
0924 16:14:25.144733 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:25.144794 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:25.144947 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:25.146131 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:25.146137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:25.146207 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:25.146384 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.146394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.147887 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.147890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.149291 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.149302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.150502 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.150509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.150601 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.150675 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 16:14:25.150743 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 16:14:25.150801 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":19,"thread_start_us":84,"threads_started":1}
I20260924 16:14:25.151327 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.145221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:42023 (local address 127.0.0.1:54024)
0924 16:14:25.145880 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:25.145894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:25.146001 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:25.146882 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:25.146885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:25.146906 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:25.146990 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.146997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.148511 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.148514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.149626 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.149636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.150892 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.150900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.151011 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.151043 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:14:25.151114 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 16:14:25.151195 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":92,"threads_started":1}
I20260924 16:14:25.151518 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.144518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.1.253.67:36577)
0924 16:14:25.145530 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:25.145545 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:25.145572 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:14:25.145685 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:25.145692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:25.145943 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:25.146126 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:25.146572 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.147599 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.149436 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.150134 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.150278 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.150427 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:25.151255 (+ 828us) server_negotiation.cc:299] Negotiation successful
0924 16:14:25.151366 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":139,"threads_started":1}
I20260924 16:14:25.152356 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.145659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.1.253.66:42023)
0924 16:14:25.146385 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:25.146398 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:25.146423 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:14:25.146511 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:25.146518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:25.146699 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:25.146884 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:25.147144 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.148340 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.149768 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.150723 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.151930 (+ 1207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.152073 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:25.152122 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 16:14:25.152206 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":118,"threads_started":1}
I20260924 16:14:25.154417 16026 tablet_service.cc:1467] Processing CreateTablet for tablet d35098c1d1fa45a3a1e574f56cd8e4c2 (DEFAULT_TABLE table=test-table [id=4086d88a721e4d53bc7b46f2607b430c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:25.154414 16297 tablet_service.cc:1467] Processing CreateTablet for tablet d35098c1d1fa45a3a1e574f56cd8e4c2 (DEFAULT_TABLE table=test-table [id=4086d88a721e4d53bc7b46f2607b430c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:25.154902 16162 tablet_service.cc:1467] Processing CreateTablet for tablet d35098c1d1fa45a3a1e574f56cd8e4c2 (DEFAULT_TABLE table=test-table [id=4086d88a721e4d53bc7b46f2607b430c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:25.155601 16297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35098c1d1fa45a3a1e574f56cd8e4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:25.155652 16026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35098c1d1fa45a3a1e574f56cd8e4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:25.155889 16162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35098c1d1fa45a3a1e574f56cd8e4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:25.162699 16381 tablet_bootstrap.cc:492] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Bootstrap starting.
I20260924 16:14:25.164227 16383 tablet_bootstrap.cc:492] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63: Bootstrap starting.
I20260924 16:14:25.165184 16381 tablet_bootstrap.cc:654] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:25.166082 16381 log.cc:824] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:25.166566 16382 tablet_bootstrap.cc:492] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8: Bootstrap starting.
I20260924 16:14:25.167255 16383 tablet_bootstrap.cc:654] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:25.168155 16383 log.cc:824] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:25.168548 16381 tablet_bootstrap.cc:492] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: No bootstrap required, opened a new log
I20260924 16:14:25.168758 16381 ts_tablet_manager.cc:1397] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:14:25.168828 16382 tablet_bootstrap.cc:654] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:25.169584 16382 log.cc:824] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:25.170560 16383 tablet_bootstrap.cc:492] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63: No bootstrap required, opened a new log
I20260924 16:14:25.170819 16383 ts_tablet_manager.cc:1397] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260924 16:14:25.171515 16382 tablet_bootstrap.cc:492] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8: No bootstrap required, opened a new log
I20260924 16:14:25.171710 16382 ts_tablet_manager.cc:1397] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:14:25.174229 16381 raft_consensus.cc:348] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } }
I20260924 16:14:25.174594 16381 raft_consensus.cc:374] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:25.174660 16381 raft_consensus.cc:729] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2353fc3ef96145278fc62b01dde95288, State: Initialized, Role: FOLLOWER
I20260924 16:14:25.175091 16382 raft_consensus.cc:348] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } }
I20260924 16:14:25.175170 16381 consensus_queue.cc:260] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } }
I20260924 16:14:25.175318 16382 raft_consensus.cc:374] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:25.175377 16382 raft_consensus.cc:729] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58776ae6a124a0da6a9c3541adba1c8, State: Initialized, Role: FOLLOWER
I20260924 16:14:25.175184 16383 raft_consensus.cc:348] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } }
I20260924 16:14:25.175442 16383 raft_consensus.cc:374] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:25.175516 16383 raft_consensus.cc:729] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbc0826a058433092d482b61f0d4b63, State: Initialized, Role: FOLLOWER
I20260924 16:14:25.175849 16382 consensus_queue.cc:260] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } }
I20260924 16:14:25.175966 16093 heartbeater.cc:503] Master 127.1.253.126:45771 was elected leader, sending a full tablet report...
I20260924 16:14:25.176015 16383 consensus_queue.cc:260] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } }
I20260924 16:14:25.176357 16381 ts_tablet_manager.cc:1428] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260924 16:14:25.176751 16382 ts_tablet_manager.cc:1428] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260924 16:14:25.176920 16383 ts_tablet_manager.cc:1428] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260924 16:14:25.177104 16364 heartbeater.cc:503] Master 127.1.253.126:45771 was elected leader, sending a full tablet report...
I20260924 16:14:25.177280 16229 heartbeater.cc:503] Master 127.1.253.126:45771 was elected leader, sending a full tablet report...
I20260924 16:14:25.203476 16387 raft_consensus.cc:482] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:14:25.203696 16387 raft_consensus.cc:504] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } }
I20260924 16:14:25.204892 16387 leader_election.cc:288] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577), bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023)
I20260924 16:14:25.209158 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.205411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:58551 (local address 127.1.253.67:36577)
0924 16:14:25.205562 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:25.205567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:25.205926 (+ 359us) server_negotiation.cc:407] Connection header received
0924 16:14:25.206087 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:25.206090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:25.206133 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:25.206219 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:25.206548 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.207046 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.208144 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.208543 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.208675 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.208730 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:25.208940 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 16:14:25.209021 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:14:25.209180 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.205160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:36577 (local address 127.1.253.65:58551)
0924 16:14:25.205821 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:25.205835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:25.205945 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:25.206250 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:25.206254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:25.206294 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:25.206414 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.206419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.207162 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.207167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.208018 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.208028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.208668 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.208672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.208802 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.208834 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:14:25.208921 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:14:25.208990 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":79,"spinlock_wait_cycles":768,"thread_start_us":136,"threads_started":1}
I20260924 16:14:25.209185 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.205394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:42421 (local address 127.1.253.66:42023)
0924 16:14:25.205534 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:25.205538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:25.205796 (+ 258us) server_negotiation.cc:407] Connection header received
0924 16:14:25.206021 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:25.206024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:25.206079 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:25.206170 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:25.206509 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.207016 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.208135 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.208537 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.208671 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.208735 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:25.208986 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 16:14:25.209060 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:14:25.209209 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.205155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:42023 (local address 127.1.253.65:42421)
0924 16:14:25.205702 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:25.205717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:25.205914 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:25.206187 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:25.206190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:25.206206 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:25.206386 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.206392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.207131 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.207137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.208007 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.208016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.208661 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.208665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.208851 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.208869 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 16:14:25.208967 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 16:14:25.209035 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":79,"spinlock_wait_cycles":9984,"thread_start_us":467,"threads_started":1}
I20260924 16:14:25.209842 16317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35098c1d1fa45a3a1e574f56cd8e4c2" candidate_uuid: "2353fc3ef96145278fc62b01dde95288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58776ae6a124a0da6a9c3541adba1c8" is_pre_election: true
I20260924 16:14:25.210186 16317 raft_consensus.cc:2454] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2353fc3ef96145278fc62b01dde95288 in term 0.
I20260924 16:14:25.210318 16182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35098c1d1fa45a3a1e574f56cd8e4c2" candidate_uuid: "2353fc3ef96145278fc62b01dde95288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbc0826a058433092d482b61f0d4b63" is_pre_election: true
I20260924 16:14:25.210616 16182 raft_consensus.cc:2454] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2353fc3ef96145278fc62b01dde95288 in term 0.
I20260924 16:14:25.210712 15982 leader_election.cc:302] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2353fc3ef96145278fc62b01dde95288, e58776ae6a124a0da6a9c3541adba1c8; no voters:
I20260924 16:14:25.211061 16387 raft_consensus.cc:2781] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:14:25.211148 16387 raft_consensus.cc:482] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:14:25.211199 16387 raft_consensus.cc:3037] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:25.212344 16387 raft_consensus.cc:504] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } }
I20260924 16:14:25.212679 16387 leader_election.cc:288] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 1 election: Requested vote from peers e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577), bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023)
I20260924 16:14:25.212986 16317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35098c1d1fa45a3a1e574f56cd8e4c2" candidate_uuid: "2353fc3ef96145278fc62b01dde95288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58776ae6a124a0da6a9c3541adba1c8"
I20260924 16:14:25.213061 16182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35098c1d1fa45a3a1e574f56cd8e4c2" candidate_uuid: "2353fc3ef96145278fc62b01dde95288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbc0826a058433092d482b61f0d4b63"
I20260924 16:14:25.213150 16317 raft_consensus.cc:3037] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:25.213183 16182 raft_consensus.cc:3037] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:25.214387 16182 raft_consensus.cc:2454] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2353fc3ef96145278fc62b01dde95288 in term 1.
I20260924 16:14:25.214388 16317 raft_consensus.cc:2454] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2353fc3ef96145278fc62b01dde95288 in term 1.
I20260924 16:14:25.214665 15981 leader_election.cc:302] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2353fc3ef96145278fc62b01dde95288, bcbc0826a058433092d482b61f0d4b63; no voters:
I20260924 16:14:25.214841 16387 raft_consensus.cc:2781] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:14:25.215052 16387 raft_consensus.cc:686] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 1 LEADER]: Becoming Leader. State: Replica: 2353fc3ef96145278fc62b01dde95288, State: Running, Role: LEADER
I20260924 16:14:25.215386 16387 consensus_queue.cc:237] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } }
I20260924 16:14:25.219254 15901 catalog_manager.cc:5495] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2353fc3ef96145278fc62b01dde95288 (127.1.253.65). New cstate: current_term: 1 leader_uuid: "2353fc3ef96145278fc62b01dde95288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20260924 16:14:25.250023 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.246176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59406 (local address 127.1.253.65:38779)
0924 16:14:25.246361 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:25.246366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:25.246380 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:14:25.246569 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:25.246571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:25.246639 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:25.246760 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:14:25.247079 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.247594 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.248510 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.248728 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.248847 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.249190 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:14:25.249619 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:14:25.249758 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:25.249816 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 16:14:25.249875 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 16:14:25.256222 16317 raft_consensus.cc:1261] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8 [term 1 FOLLOWER]: Refusing update from remote peer 2353fc3ef96145278fc62b01dde95288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:14:25.256222 16182 raft_consensus.cc:1261] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63 [term 1 FOLLOWER]: Refusing update from remote peer 2353fc3ef96145278fc62b01dde95288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:14:25.256994 16387 consensus_queue.cc:1035] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:14:25.257217 16392 consensus_queue.cc:1035] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:14:25.266484 16397 mvcc.cc:205] Tried to move back new op lower bound from 7332931441681883136 to 7332931441525149696. Current Snapshot: MvccSnapshot[applied={T|T < 7332931441681883136}]
I20260924 16:14:25.274935 15899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:14:25.275153 15899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:14:25.277976 15899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Sending DeleteTablet for 3 replicas of tablet d35098c1d1fa45a3a1e574f56cd8e4c2
I20260924 16:14:25.278934 16026 tablet_service.cc:1514] Processing DeleteTablet for tablet d35098c1d1fa45a3a1e574f56cd8e4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:25 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:59400
I20260924 16:14:25.278923 16297 tablet_service.cc:1514] Processing DeleteTablet for tablet d35098c1d1fa45a3a1e574f56cd8e4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:25 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:56398
I20260924 16:14:25.279282 16162 tablet_service.cc:1514] Processing DeleteTablet for tablet d35098c1d1fa45a3a1e574f56cd8e4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:25 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:54024
I20260924 16:14:25.279600 16406 tablet_replica.cc:330] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: stopping tablet replica
I20260924 16:14:25.279636 16405 tablet_replica.cc:330] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8: stopping tablet replica
I20260924 16:14:25.279736 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 15963
I20260924 16:14:25.281774 16405 raft_consensus.cc:2229] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:14:25.281903 16407 tablet_replica.cc:330] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63: stopping tablet replica
I20260924 16:14:25.282198 16405 raft_consensus.cc:2258] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:25.282213 16407 raft_consensus.cc:2229] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:14:25.282545 16407 raft_consensus.cc:2258] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:25.284217 16405 ts_tablet_manager.cc:1905] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:14:25.284375 16407 ts_tablet_manager.cc:1905] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:14:25.287114 16407 ts_tablet_manager.cc:1918] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:14:25.287114 16405 ts_tablet_manager.cc:1918] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:14:25.287225 16407 log.cc:1196] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/d35098c1d1fa45a3a1e574f56cd8e4c2
I20260924 16:14:25.287225 16405 log.cc:1196] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/d35098c1d1fa45a3a1e574f56cd8e4c2
I20260924 16:14:25.287585 16407 ts_tablet_manager.cc:1939] T d35098c1d1fa45a3a1e574f56cd8e4c2 P bcbc0826a058433092d482b61f0d4b63: Deleting consensus metadata
I20260924 16:14:25.287585 16405 ts_tablet_manager.cc:1939] T d35098c1d1fa45a3a1e574f56cd8e4c2 P e58776ae6a124a0da6a9c3541adba1c8: Deleting consensus metadata
W20260924 16:14:25.288910 15886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:38779 (error 108)
I20260924 16:14:25.288928 15888 catalog_manager.cc:4841] TS e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577): tablet d35098c1d1fa45a3a1e574f56cd8e4c2 (table test-table [id=4086d88a721e4d53bc7b46f2607b430c]) successfully deleted
I20260924 16:14:25.288932 15887 catalog_manager.cc:4841] TS bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023): tablet d35098c1d1fa45a3a1e574f56cd8e4c2 (table test-table [id=4086d88a721e4d53bc7b46f2607b430c]) successfully deleted
I20260924 16:14:25.289896 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 16098
I20260924 16:14:25.290017 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.289691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:38779 (local address 127.0.0.1:59416)
0924 16:14:25.289844 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:25.289906 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:14:25.291019 15886 catalog_manager.cc:4584] TS 2353fc3ef96145278fc62b01dde95288 (127.1.253.65:38779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d35098c1d1fa45a3a1e574f56cd8e4c2: Network error: Client connection negotiation failed: client connection to 127.1.253.65:38779: connect: Connection refused (error 111)
I20260924 16:14:25.296530 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 16234
I20260924 16:14:25.303697 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 15869
I20260924 16:14:25.310848 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:45771
--webserver_interface=127.1.253.126
--webserver_port=41869
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:45771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:25.425150 16409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:25.425467 16409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:25.425549 16409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:25.429672 16409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:14:25.429773 16409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:25.429811 16409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:25.429870 16409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:14:25.429925 16409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:14:25.435976 16409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:45771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:45771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=41869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:25.437147 16409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:25.438268 16409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:14:25.453195 16409 server_base.cc:1034] running on GCE node
W20260924 16:14:25.453147 16416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:25.453147 16415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:25.453148 16418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:25.454032 16409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:25.455323 16409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:25.456516 16409 hybrid_clock.cc:648] HybridClock initialized: now 1790266465456494 us; error 35 us; skew 500 ppm
Sep 24 16:14:25 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:25.459537 16409 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:14:25.460561 16409 webserver.cc:460] Webserver started at http://127.1.253.126:41869/ using document root <none> and password file <none>
I20260924 16:14:25.461181 16409 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:25.461297 16409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:25.465359 16409 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 16:14:25.467661 16425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:25.468942 16409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:14:25.469094 16409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "76876cc5fc2d4c7fb62f865db014fe49"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "2f01d4ff54eb8b8087459ebb76d5e933"
server_key_iv: "d5dc6962a82f5e34b844ede353e3a743"
server_key_version: "encryptionkey@0"
I20260924 16:14:25.469722 16409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:25.483884 16409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:25.487452 16409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:25.487694 16409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:25.495491 16409 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:45771
I20260924 16:14:25.495793 16477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:45771 every 8 connection(s)
I20260924 16:14:25.496767 16409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:14:25.498047 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 16409
I20260924 16:14:25.498600 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:38779
--local_ip_for_outbound_sockets=127.1.253.65
--tserver_master_addrs=127.1.253.126:45771
--webserver_port=41429
--webserver_interface=127.1.253.65
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:14:25.502859 16478 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:14:25.504736 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Bootstrap starting.
I20260924 16:14:25.523155 16478 log.cc:824] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:25.532356 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:14:25.532747 16478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Bootstrap complete.
I20260924 16:14:25.537052 16478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } }
I20260924 16:14:25.537487 16478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76876cc5fc2d4c7fb62f865db014fe49, State: Initialized, Role: FOLLOWER
I20260924 16:14:25.538034 16478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } }
I20260924 16:14:25.538180 16478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:14:25.538296 16478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:14:25.538422 16478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:14:25.540380 16478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } }
I20260924 16:14:25.540745 16478 leader_election.cc:302] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76876cc5fc2d4c7fb62f865db014fe49; no voters:
I20260924 16:14:25.541105 16478 leader_election.cc:288] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:14:25.541193 16482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:14:25.541405 16482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 2 LEADER]: Becoming Leader. State: Replica: 76876cc5fc2d4c7fb62f865db014fe49, State: Running, Role: LEADER
I20260924 16:14:25.541702 16482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } }
I20260924 16:14:25.542451 16478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:14:25.543241 16483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "76876cc5fc2d4c7fb62f865db014fe49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } } }
I20260924 16:14:25.543397 16483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:25.543900 16484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76876cc5fc2d4c7fb62f865db014fe49. Latest consensus state: current_term: 2 leader_uuid: "76876cc5fc2d4c7fb62f865db014fe49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } } }
I20260924 16:14:25.544046 16484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:25.544342 16491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:14:25.549476 16491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:14:25.550096 16491 catalog_manager.cc:1259] Loaded cluster ID: 301e9ca6e8634dffba066a28ae2305ef
I20260924 16:14:25.550158 16491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:14:25.551559 16491 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 16:14:25.556046 16491 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 16:14:25.561894 16491 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:14:25.562826 16491 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Loaded TSK: 0
I20260924 16:14:25.564155 16491 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:14:25.622567 16480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:25.622853 16480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:25.622925 16480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:25.627175 16480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:25.627239 16480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:25.627300 16480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:14:25.632781 16480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:38779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=41429
--tserver_master_addrs=127.1.253.126:45771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:25.633855 16480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:25.635061 16480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:25.646389 16507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:25.646375 16506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:25.646457 16509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:25.646880 16480 server_base.cc:1034] running on GCE node
I20260924 16:14:25.647342 16480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:25.648036 16480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:25.649227 16480 hybrid_clock.cc:648] HybridClock initialized: now 1790266465649196 us; error 47 us; skew 500 ppm
Sep 24 16:14:25 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:25.652209 16480 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:14:25.653129 16480 webserver.cc:460] Webserver started at http://127.1.253.65:41429/ using document root <none> and password file <none>
I20260924 16:14:25.653709 16480 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:25.653813 16480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:25.657966 16480 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:14:25.660408 16516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:25.662071 16480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:14:25.662197 16480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "2353fc3ef96145278fc62b01dde95288"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "d3922a3f99782138e901eebc45376e0c"
server_key_iv: "af11e315ca356a5a7160c0507cacd9d3"
server_key_version: "encryptionkey@0"
I20260924 16:14:25.662882 16480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:25.691583 16480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:25.694841 16480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:25.695070 16480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:25.695711 16480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:25.696971 16523 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 16:14:25.701407 16480 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:14:25.701496 16480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 16:14:25.701582 16480 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:14:25.703435 16480 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:14:25.703511 16480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260924 16:14:25.703802 16523 tablet_bootstrap.cc:492] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Bootstrap starting.
I20260924 16:14:25.714421 16480 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:38779
I20260924 16:14:25.714506 16630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:38779 every 8 connection(s)
I20260924 16:14:25.715543 16480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:14:25.716621 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 16480
I20260924 16:14:25.717061 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:42023
--local_ip_for_outbound_sockets=127.1.253.66
--tserver_master_addrs=127.1.253.126:45771
--webserver_port=41847
--webserver_interface=127.1.253.66
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:14:25.723805 16523 log.cc:824] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:14:25 dist-test-slave-6wvm krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:25.733117 16523 tablet_bootstrap.cc:492] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:14:25.733609 16523 tablet_bootstrap.cc:492] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Bootstrap complete.
I20260924 16:14:25.735085 16523 ts_tablet_manager.cc:1397] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.011s
I20260924 16:14:25.738313 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.718560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:49877 (local address 127.1.253.126:45771)
0924 16:14:25.721095 (+ 2535us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:25.721119 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:25.721176 (+ 57us) server_negotiation.cc:407] Connection header received
0924 16:14:25.721575 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:25.721592 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:25.722060 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:25.722521 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:25.724229 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.725981 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.727700 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.728574 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.728790 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.732176 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:25.732183 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:25.732199 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:25.732234 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:25.734597 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:25.735256 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:25.735262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:25.735268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:25.735419 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:25.735827 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:25.735830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:25.735832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:25.736309 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:25.736441 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:25.737118 (+ 677us) server_negotiation.cc:299] Negotiation successful
0924 16:14:25.737439 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2326,"thread_start_us":123,"threads_started":1}
I20260924 16:14:25.738008 16523 raft_consensus.cc:348] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } }
I20260924 16:14:25.738561 16523 raft_consensus.cc:729] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2353fc3ef96145278fc62b01dde95288, State: Initialized, Role: FOLLOWER
I20260924 16:14:25.738336 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.718436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:49877)
0924 16:14:25.719027 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:25.719086 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:25.719964 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:25.722628 (+ 2664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:25.722641 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:25.723220 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:25.723946 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.723973 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.726188 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.726192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.727536 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.727548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.728762 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.728767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.728967 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.729714 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:25.729735 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:25.731909 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:25.734729 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:25.734736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:25.734750 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:25.735111 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:25.735529 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:25.735533 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:25.735534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:25.735707 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:25.736441 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:25.736448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:25.736738 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 16:14:25.736947 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 16:14:25.737301 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":177,"threads_started":1}
I20260924 16:14:25.739753 16523 consensus_queue.cc:260] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } }
I20260924 16:14:25.740913 16523 ts_tablet_manager.cc:1428] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:14:25.741626 16631 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45771
I20260924 16:14:25.741870 16631 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:25.742484 16631 heartbeater.cc:511] Master 127.1.253.126:45771 requested a full tablet report, sending...
I20260924 16:14:25.745030 16442 ts_manager.cc:194] Registered new tserver with Master: 2353fc3ef96145278fc62b01dde95288 (127.1.253.65:38779)
I20260924 16:14:25.748867 16442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:49877
I20260924 16:14:25.751819 16631 heartbeater.cc:503] Master 127.1.253.126:45771 was elected leader, sending a full tablet report...
Sep 24 16:14:25 dist-test-slave-6wvm krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.65@KRBTEST.COM
I20260924 16:14:25.759562 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.747977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:38779 (local address 127.0.0.1:59426)
0924 16:14:25.748455 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:25.748500 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:25.748629 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:25.749276 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:25.749281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:25.749750 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:25.749882 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.749891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.751075 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.751081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.751747 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.751758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.753324 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.753327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.753457 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.754071 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:25.754084 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:25.755694 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:25.757624 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:25.757634 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:25.757640 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:25.757891 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:25.758224 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:25.758227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:25.758229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:25.758357 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:25.758856 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:25.758863 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:25.758995 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 16:14:25.759142 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 16:14:25.759365 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":124,"threads_started":1}
I20260924 16:14:25.759572 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.748051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.1.253.65:38779)
0924 16:14:25.748781 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:25.748793 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:25.748817 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:14:25.748886 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:25.748891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:25.749111 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:25.749253 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:25.750049 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.750952 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.751953 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.752417 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.752538 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.755847 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:25.755858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:25.755877 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:25.755901 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:25.757498 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:25.758026 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:25.758032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:25.758033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:25.758114 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:25.758452 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:25.758455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:25.758457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:25.758717 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:25.758871 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:25.759238 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 16:14:25.759418 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":633,"thread_start_us":108,"threads_started":1}
I20260924 16:14:25.760764 16565 tablet_service.cc:1514] Processing DeleteTablet for tablet d35098c1d1fa45a3a1e574f56cd8e4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:25 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:59426
I20260924 16:14:25.761884 16640 tablet_replica.cc:330] stopping tablet replica
I20260924 16:14:25.762207 16640 raft_consensus.cc:2229] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:14:25.762534 16640 raft_consensus.cc:2258] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:25.763579 16640 ts_tablet_manager.cc:1905] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:14:25.767710 16640 ts_tablet_manager.cc:1918] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:14:25.767863 16640 log.cc:1196] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal/wals/d35098c1d1fa45a3a1e574f56cd8e4c2
I20260924 16:14:25.768142 16640 ts_tablet_manager.cc:1939] T d35098c1d1fa45a3a1e574f56cd8e4c2 P 2353fc3ef96145278fc62b01dde95288: Deleting consensus metadata
I20260924 16:14:25.770056 16427 catalog_manager.cc:4841] TS 2353fc3ef96145278fc62b01dde95288 (127.1.253.65:38779): tablet d35098c1d1fa45a3a1e574f56cd8e4c2 (table test-table [id=4086d88a721e4d53bc7b46f2607b430c]) successfully deleted
W20260924 16:14:25.846655 16633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:25.846915 16633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:25.846987 16633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:25.851032 16633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:25.851083 16633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:25.851167 16633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:14:25.856386 16633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:42023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=41847
--tserver_master_addrs=127.1.253.126:45771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:25.857486 16633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:25.858619 16633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:25.869381 16646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:25.869359 16633 server_base.cc:1034] running on GCE node
W20260924 16:14:25.869333 16648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:25.869333 16645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:25.870033 16633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:25.870780 16633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:25.871971 16633 hybrid_clock.cc:648] HybridClock initialized: now 1790266465871943 us; error 56 us; skew 500 ppm
Sep 24 16:14:25 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:25.874821 16633 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:14:25.875747 16633 webserver.cc:460] Webserver started at http://127.1.253.66:41847/ using document root <none> and password file <none>
I20260924 16:14:25.876343 16633 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:25.876471 16633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:25.880486 16633 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:14:25.882597 16655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:25.883908 16633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:14:25.884075 16633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "bcbc0826a058433092d482b61f0d4b63"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "c7edb68bf1c0010bcb9c8a942dc11073"
server_key_iv: "69d56ec57fced7665a39b654c3c2183a"
server_key_version: "encryptionkey@0"
I20260924 16:14:25.884655 16633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:25.904018 16633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:25.907197 16633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:25.907428 16633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:25.908080 16633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:25.908965 16633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:25.909039 16633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:25.909121 16633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:25.909170 16633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:25.919111 16633 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:42023
I20260924 16:14:25.919169 16770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:42023 every 8 connection(s)
I20260924 16:14:25.920233 16633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:14:25.927937 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 16633
I20260924 16:14:25.928440 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:36577
--local_ip_for_outbound_sockets=127.1.253.67
--tserver_master_addrs=127.1.253.126:45771
--webserver_port=41837
--webserver_interface=127.1.253.67
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:14:25 dist-test-slave-6wvm krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:25.935972 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.922360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:51143 (local address 127.1.253.126:45771)
0924 16:14:25.922525 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:25.922530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:25.923696 (+ 1166us) server_negotiation.cc:407] Connection header received
0924 16:14:25.924554 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:25.924557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:25.924611 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:25.924689 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:25.925942 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.926606 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.927861 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.928170 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.928251 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.931678 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:25.931682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:25.931684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:25.931713 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:25.933383 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:25.934105 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:25.934108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:25.934109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:25.934190 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:25.934682 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:25.934684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:25.934685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:25.934877 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:25.934954 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:25.935720 (+ 766us) server_negotiation.cc:299] Negotiation successful
0924 16:14:25.935842 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:14:25.937445 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:25.922835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:51143)
0924 16:14:25.923516 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:25.923579 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:25.924361 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:25.924867 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:25.924874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:25.925263 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:25.925776 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.925790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.926729 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.926732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.927700 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:25.927715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:25.928336 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:25.928344 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:25.928561 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:25.929231 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:25.929257 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:25.931468 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:25.933526 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:25.933538 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:25.933562 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:25.933997 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:25.934333 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:25.934338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:25.934343 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:25.934568 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:25.934992 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:25.935002 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:25.935421 (+ 419us) client_negotiation.cc:769] Sending connection context
0924 16:14:25.935692 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 16:14:25.936140 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":108,"threads_started":1}
I20260924 16:14:25.938863 16771 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45771
I20260924 16:14:25.939146 16771 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:25.939689 16771 heartbeater.cc:511] Master 127.1.253.126:45771 requested a full tablet report, sending...
I20260924 16:14:25.940907 16442 ts_manager.cc:194] Registered new tserver with Master: bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023)
I20260924 16:14:25.941511 16442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:51143
W20260924 16:14:26.047331 16774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:26.047639 16774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:26.047724 16774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:26.051908 16774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:26.051995 16774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:26.052085 16774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:14:26.057507 16774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:36577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=41837
--tserver_master_addrs=127.1.253.126:45771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:26.058732 16774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:26.059870 16774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:26.070389 16779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:26.070327 16780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:26.070333 16782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:26.070501 16774 server_base.cc:1034] running on GCE node
I20260924 16:14:26.071064 16774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:26.071683 16774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:26.072860 16774 hybrid_clock.cc:648] HybridClock initialized: now 1790266466072840 us; error 28 us; skew 500 ppm
Sep 24 16:14:26 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266466, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:26.075963 16774 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:14:26.076889 16774 webserver.cc:460] Webserver started at http://127.1.253.67:41837/ using document root <none> and password file <none>
I20260924 16:14:26.077482 16774 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:26.077597 16774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:26.081588 16774 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:14:26.085542 16789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:26.087507 16774 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:14:26.087659 16774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "e58776ae6a124a0da6a9c3541adba1c8"
format_stamp: "Formatted at 2026-09-24 16:14:25 on dist-test-slave-6wvm"
server_key: "982841ad1648d7fd12ae7c79fb2f49de"
server_key_iv: "8802374937963b8def61c05ab136dd8b"
server_key_version: "encryptionkey@0"
I20260924 16:14:26.088548 16774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:26.100935 16774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:26.103963 16774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:26.104182 16774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:26.104830 16774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:26.105731 16774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:26.105801 16774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:26.105876 16774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:26.105928 16774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:26.116128 16774 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:36577
I20260924 16:14:26.116148 16903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:36577 every 8 connection(s)
I20260924 16:14:26.117231 16774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:14:26.126582 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 16774
Sep 24 16:14:26 dist-test-slave-6wvm krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266466, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:26.132047 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.119365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:40109 (local address 127.1.253.126:45771)
0924 16:14:26.119522 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:26.119526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:26.120389 (+ 863us) server_negotiation.cc:407] Connection header received
0924 16:14:26.121804 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:26.121808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:26.121908 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:26.122015 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:26.123264 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.123798 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.125000 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.125236 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.125343 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.128281 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:26.128285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:26.128288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:26.128314 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:26.129856 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:26.130454 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:26.130457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:26.130459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:26.130545 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:26.130897 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:26.130901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:26.130902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:26.131082 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:26.131190 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:26.131728 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 16:14:26.131863 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:14:26.133124 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.119738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:40109)
0924 16:14:26.120203 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:26.120247 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:26.121580 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:26.122211 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:26.122219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:26.122619 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:26.123106 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:26.123119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.123942 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.123946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:26.124862 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:26.124871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.125411 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.125416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:26.125584 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.126224 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:26.126294 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:26.128110 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:26.129987 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:26.129993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:26.130008 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:26.130335 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:26.130639 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:26.130642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:26.130644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:26.130793 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:26.131191 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:26.131198 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:26.131478 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 16:14:26.131671 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 16:14:26.132011 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260924 16:14:26.134552 16904 heartbeater.cc:347] Connected to a master server at 127.1.253.126:45771
I20260924 16:14:26.134831 16904 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:26.135337 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.127369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.1.253.126:45771)
0924 16:14:26.127650 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:26.127655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:26.127721 (+ 66us) server_negotiation.cc:407] Connection header received
0924 16:14:26.127893 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:26.127896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:26.127949 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:26.128046 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:26.128775 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.129305 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.130165 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.130461 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.130579 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.131895 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:26.131899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:26.131901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:26.131934 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:26.133528 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:26.134003 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:26.134010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:26.134014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:26.134109 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:26.134429 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:26.134431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:26.134432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:26.134595 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:26.134814 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:26.135055 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 16:14:26.135181 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":104,"threads_started":1}
I20260924 16:14:26.135452 16904 heartbeater.cc:511] Master 127.1.253.126:45771 requested a full tablet report, sending...
I20260924 16:14:26.136535 16442 ts_manager.cc:194] Registered new tserver with Master: e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577)
I20260924 16:14:26.137095 16442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:40109
I20260924 16:14:26.137516 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:14:26.142808 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.138188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.1.253.126:45771)
0924 16:14:26.138335 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:26.138339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:26.139304 (+ 965us) server_negotiation.cc:407] Connection header received
0924 16:14:26.139489 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:26.139492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:26.139579 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:26.139685 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:14:26.139919 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.140412 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.141278 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.141506 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.141622 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.141934 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:14:26.142438 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:14:26.142573 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:26.142633 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 16:14:26.142689 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:14:26.143481 16442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:14:26.147080 16442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:14:26.155228 16565 tablet_service.cc:1467] Processing CreateTablet for tablet bb2e00834a614bdf943095bb1c94f0da (DEFAULT_TABLE table=test-table [id=9c1c9bd78130448985b06bea3ec05ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:26.155980 16565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2e00834a614bdf943095bb1c94f0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:26.158975 16523 tablet_bootstrap.cc:492] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: Bootstrap starting.
I20260924 16:14:26.159443 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.154062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:36577 (local address 127.0.0.1:56414)
0924 16:14:26.154218 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:26.154294 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:26.154406 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:26.155476 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:26.155480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:26.155501 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:26.155610 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:26.155618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.156975 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.156979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:26.157801 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:26.157810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.158957 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.158961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:26.159068 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.159119 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 16:14:26.159208 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 16:14:26.159298 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 16:14:26.159502 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.154447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56414 (local address 127.1.253.67:36577)
0924 16:14:26.154912 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:26.154925 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:26.154948 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:14:26.155041 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:26.155047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:26.155282 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:26.155464 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:26.155751 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.156832 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.157943 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.158816 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.158922 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.159104 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:26.159241 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 16:14:26.159349 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":123,"threads_started":1}
I20260924 16:14:26.159969 16523 tablet_bootstrap.cc:654] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:26.161634 16523 tablet_bootstrap.cc:492] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: No bootstrap required, opened a new log
I20260924 16:14:26.161734 16523 ts_tablet_manager.cc:1397] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 16:14:26.162199 16523 raft_consensus.cc:348] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:26.162366 16523 raft_consensus.cc:374] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:26.162413 16523 raft_consensus.cc:729] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2353fc3ef96145278fc62b01dde95288, State: Initialized, Role: FOLLOWER
I20260924 16:14:26.162578 16523 consensus_queue.cc:260] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:26.162933 16837 tablet_service.cc:1467] Processing CreateTablet for tablet bb2e00834a614bdf943095bb1c94f0da (DEFAULT_TABLE table=test-table [id=9c1c9bd78130448985b06bea3ec05ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:26.164168 16837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2e00834a614bdf943095bb1c94f0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:26.164491 16523 ts_tablet_manager.cc:1428] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 16:14:26.166765 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.159838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:42023 (local address 127.0.0.1:54026)
0924 16:14:26.160081 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:26.160095 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:26.160185 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:26.162734 (+ 2549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:26.162737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:26.162760 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:26.162857 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:26.162864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.164266 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.164270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:26.165312 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:26.165321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.166374 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.166378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:26.166464 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.166488 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:14:26.166555 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 16:14:26.166639 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260924 16:14:26.166834 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.159431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.1.253.66:42023)
0924 16:14:26.159983 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:26.159998 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:26.160187 (+ 189us) server_negotiation.cc:407] Connection header received
0924 16:14:26.160291 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:26.160298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:26.162576 (+ 2278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:26.162935 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:26.163124 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.164126 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.165439 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.166217 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.166377 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.166547 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:26.166602 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 16:14:26.166689 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":163,"threads_started":1}
I20260924 16:14:26.170044 16705 tablet_service.cc:1467] Processing CreateTablet for tablet bb2e00834a614bdf943095bb1c94f0da (DEFAULT_TABLE table=test-table [id=9c1c9bd78130448985b06bea3ec05ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:26.171140 16705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2e00834a614bdf943095bb1c94f0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:26.172111 16916 tablet_bootstrap.cc:492] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8: Bootstrap starting.
I20260924 16:14:26.174569 16916 tablet_bootstrap.cc:654] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:26.175901 16916 log.cc:824] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:26.178190 16916 tablet_bootstrap.cc:492] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8: No bootstrap required, opened a new log
I20260924 16:14:26.178448 16916 ts_tablet_manager.cc:1397] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260924 16:14:26.179870 16918 tablet_bootstrap.cc:492] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63: Bootstrap starting.
I20260924 16:14:26.182098 16916 raft_consensus.cc:348] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:26.182432 16916 raft_consensus.cc:374] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:26.182500 16916 raft_consensus.cc:729] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58776ae6a124a0da6a9c3541adba1c8, State: Initialized, Role: FOLLOWER
I20260924 16:14:26.182632 16918 tablet_bootstrap.cc:654] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:26.182966 16916 consensus_queue.cc:260] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:26.183555 16918 log.cc:824] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:26.183715 16904 heartbeater.cc:503] Master 127.1.253.126:45771 was elected leader, sending a full tablet report...
I20260924 16:14:26.183967 16916 ts_tablet_manager.cc:1428] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260924 16:14:26.185580 16918 tablet_bootstrap.cc:492] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63: No bootstrap required, opened a new log
I20260924 16:14:26.185825 16918 ts_tablet_manager.cc:1397] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 16:14:26.189215 16918 raft_consensus.cc:348] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:26.189538 16918 raft_consensus.cc:374] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:26.189636 16918 raft_consensus.cc:729] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbc0826a058433092d482b61f0d4b63, State: Initialized, Role: FOLLOWER
I20260924 16:14:26.190093 16918 consensus_queue.cc:260] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:26.190788 16771 heartbeater.cc:503] Master 127.1.253.126:45771 was elected leader, sending a full tablet report...
I20260924 16:14:26.191068 16918 ts_tablet_manager.cc:1428] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260924 16:14:26.216805 16632 tablet.cc:2369] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:14:26.368263 16905 tablet.cc:2369] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:14:26.421669 16772 tablet.cc:2369] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:14:26.493108 16921 raft_consensus.cc:482] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:14:26.493320 16921 raft_consensus.cc:504] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:26.494647 16921 leader_election.cc:288] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577), 2353fc3ef96145278fc62b01dde95288 (127.1.253.65:38779)
I20260924 16:14:26.499614 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.495126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:57255 (local address 127.1.253.65:38779)
0924 16:14:26.495427 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:26.495432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:26.495666 (+ 234us) server_negotiation.cc:407] Connection header received
0924 16:14:26.495761 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:26.495765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:26.495854 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:26.495972 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:26.496306 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.496871 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.498334 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.498995 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.499135 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.499299 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:26.499386 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 16:14:26.499466 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
I20260924 16:14:26.499639 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.495010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:38779 (local address 127.1.253.66:57255)
0924 16:14:26.495436 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:26.495452 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:26.495586 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:26.496004 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:26.496007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:26.496026 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:26.496158 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:26.496165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.496995 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.497000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:26.498171 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:26.498181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.499141 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.499145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:26.499259 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.499284 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:14:26.499373 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 16:14:26.499468 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":94,"threads_started":1}
I20260924 16:14:26.499728 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.495219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:33363 (local address 127.1.253.67:36577)
0924 16:14:26.495474 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:26.495481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:26.495730 (+ 249us) server_negotiation.cc:407] Connection header received
0924 16:14:26.495856 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:26.495860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:26.495915 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:26.495989 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:26.496404 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.497254 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.498599 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.499011 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.499153 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.499209 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:26.499519 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 16:14:26.499597 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260924 16:14:26.500152 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.495010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:36577 (local address 127.1.253.66:33363)
0924 16:14:26.495599 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:26.495613 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:26.495746 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:26.496114 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:26.496117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:26.496132 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:26.496271 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:26.496279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.497388 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.497392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:26.498479 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:26.498487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.499276 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.499279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:26.499391 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.499410 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 16:14:26.499940 (+ 530us) client_negotiation.cc:240] Negotiation successful
0924 16:14:26.500008 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":56,"thread_start_us":65,"threads_started":1}
I20260924 16:14:26.500669 16585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2e00834a614bdf943095bb1c94f0da" candidate_uuid: "bcbc0826a058433092d482b61f0d4b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2353fc3ef96145278fc62b01dde95288" is_pre_election: true
I20260924 16:14:26.500864 16858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2e00834a614bdf943095bb1c94f0da" candidate_uuid: "bcbc0826a058433092d482b61f0d4b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58776ae6a124a0da6a9c3541adba1c8" is_pre_election: true
I20260924 16:14:26.501034 16585 raft_consensus.cc:2454] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcbc0826a058433092d482b61f0d4b63 in term 0.
I20260924 16:14:26.501150 16858 raft_consensus.cc:2454] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcbc0826a058433092d482b61f0d4b63 in term 0.
I20260924 16:14:26.501590 16657 leader_election.cc:302] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2353fc3ef96145278fc62b01dde95288, bcbc0826a058433092d482b61f0d4b63; no voters:
I20260924 16:14:26.501926 16921 raft_consensus.cc:2781] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:14:26.502033 16921 raft_consensus.cc:482] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:14:26.502091 16921 raft_consensus.cc:3037] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:26.503232 16921 raft_consensus.cc:504] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:26.503672 16921 leader_election.cc:288] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [CANDIDATE]: Term 1 election: Requested vote from peers e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577), 2353fc3ef96145278fc62b01dde95288 (127.1.253.65:38779)
I20260924 16:14:26.504068 16858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2e00834a614bdf943095bb1c94f0da" candidate_uuid: "bcbc0826a058433092d482b61f0d4b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58776ae6a124a0da6a9c3541adba1c8"
I20260924 16:14:26.504067 16585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2e00834a614bdf943095bb1c94f0da" candidate_uuid: "bcbc0826a058433092d482b61f0d4b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2353fc3ef96145278fc62b01dde95288"
I20260924 16:14:26.504199 16858 raft_consensus.cc:3037] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:26.504235 16585 raft_consensus.cc:3037] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:26.505259 16585 raft_consensus.cc:2454] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcbc0826a058433092d482b61f0d4b63 in term 1.
I20260924 16:14:26.505520 16858 raft_consensus.cc:2454] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcbc0826a058433092d482b61f0d4b63 in term 1.
I20260924 16:14:26.505621 16657 leader_election.cc:302] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2353fc3ef96145278fc62b01dde95288, bcbc0826a058433092d482b61f0d4b63; no voters:
I20260924 16:14:26.505842 16921 raft_consensus.cc:2781] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:14:26.506091 16921 raft_consensus.cc:686] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 1 LEADER]: Becoming Leader. State: Replica: bcbc0826a058433092d482b61f0d4b63, State: Running, Role: LEADER
I20260924 16:14:26.506698 16921 consensus_queue.cc:237] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:26.509979 16439 catalog_manager.cc:5495] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcbc0826a058433092d482b61f0d4b63 (127.1.253.66). New cstate: current_term: 1 leader_uuid: "bcbc0826a058433092d482b61f0d4b63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } health_report { overall_health: UNKNOWN } } }
I20260924 16:14:26.532311 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.528595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.1.253.66:42023)
0924 16:14:26.528745 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:26.528749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:26.528763 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:14:26.528868 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:26.528870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:26.528930 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:26.529029 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:14:26.529297 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.529870 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.530789 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:26.531017 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:26.531109 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:26.531433 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:14:26.531898 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:14:26.532009 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:26.532141 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 16:14:26.532197 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:14:26.538723 16585 raft_consensus.cc:1261] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Refusing update from remote peer bcbc0826a058433092d482b61f0d4b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:14:26.538719 16858 raft_consensus.cc:1261] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8 [term 1 FOLLOWER]: Refusing update from remote peer bcbc0826a058433092d482b61f0d4b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:14:26.539522 16925 consensus_queue.cc:1035] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:14:26.539700 16921 consensus_queue.cc:1035] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:14:26.555382 16439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:14:26.555591 16439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:14:26.558295 16439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Sending DeleteTablet for 3 replicas of tablet bb2e00834a614bdf943095bb1c94f0da
I20260924 16:14:26.559146 16565 tablet_service.cc:1514] Processing DeleteTablet for tablet bb2e00834a614bdf943095bb1c94f0da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:26 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:59426
I20260924 16:14:26.559119 16837 tablet_service.cc:1514] Processing DeleteTablet for tablet bb2e00834a614bdf943095bb1c94f0da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:26 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:56414
I20260924 16:14:26.559121 16705 tablet_service.cc:1514] Processing DeleteTablet for tablet bb2e00834a614bdf943095bb1c94f0da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:26 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:54026
I20260924 16:14:26.559669 16939 tablet_replica.cc:330] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8: stopping tablet replica
I20260924 16:14:26.559762 16938 tablet_replica.cc:330] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: stopping tablet replica
I20260924 16:14:26.559913 16938 raft_consensus.cc:2229] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:14:26.559978 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 16480
I20260924 16:14:26.560097 16938 raft_consensus.cc:2258] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:26.561808 16940 tablet_replica.cc:330] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63: stopping tablet replica
I20260924 16:14:26.562156 16940 raft_consensus.cc:2229] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:14:26.562587 16940 raft_consensus.cc:2258] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:26.564472 16940 ts_tablet_manager.cc:1905] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:14:26.560097 16939 raft_consensus.cc:2229] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:14:26.565594 16939 raft_consensus.cc:2258] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:26.567653 16940 ts_tablet_manager.cc:1918] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:14:26.567755 16940 log.cc:1196] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/bb2e00834a614bdf943095bb1c94f0da
I20260924 16:14:26.568084 16940 ts_tablet_manager.cc:1939] T bb2e00834a614bdf943095bb1c94f0da P bcbc0826a058433092d482b61f0d4b63: Deleting consensus metadata
I20260924 16:14:26.568074 16939 ts_tablet_manager.cc:1905] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:14:26.569361 16428 catalog_manager.cc:4841] TS bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023): tablet bb2e00834a614bdf943095bb1c94f0da (table test-table [id=9c1c9bd78130448985b06bea3ec05ae4]) successfully deleted
W20260924 16:14:26.570034 16427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:38779 (error 108)
I20260924 16:14:26.570331 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 16633
I20260924 16:14:26.572082 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:26.571494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:38779 (local address 127.0.0.1:48400)
0924 16:14:26.571908 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:26.571990 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301}
W20260924 16:14:26.572469 16427 catalog_manager.cc:4584] TS 2353fc3ef96145278fc62b01dde95288 (127.1.253.65:38779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb2e00834a614bdf943095bb1c94f0da: Network error: Client connection negotiation failed: client connection to 127.1.253.65:38779: connect: Connection refused (error 111)
I20260924 16:14:26.572794 16939 ts_tablet_manager.cc:1918] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:14:26.572955 16939 log.cc:1196] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/bb2e00834a614bdf943095bb1c94f0da
I20260924 16:14:26.573267 16939 ts_tablet_manager.cc:1939] T bb2e00834a614bdf943095bb1c94f0da P e58776ae6a124a0da6a9c3541adba1c8: Deleting consensus metadata
I20260924 16:14:26.574585 16429 catalog_manager.cc:4841] TS e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577): tablet bb2e00834a614bdf943095bb1c94f0da (table test-table [id=9c1c9bd78130448985b06bea3ec05ae4]) successfully deleted
I20260924 16:14:26.577687 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 16774
I20260924 16:14:26.584059 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 16409
I20260924 16:14:26.591205 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:45771
--webserver_interface=127.1.253.126
--webserver_port=41869
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:45771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:26.707235 16942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:26.707603 16942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:26.707794 16942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:26.712109 16942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:14:26.712231 16942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:26.712273 16942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:26.712313 16942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:14:26.712347 16942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:14:26.718127 16942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:45771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:45771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=41869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:26.719419 16942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:26.720701 16942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:26.730069 16950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:26.730060 16947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:26.730206 16942 server_base.cc:1034] running on GCE node
W20260924 16:14:26.730069 16948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:26.730918 16942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:26.732116 16942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:26.733317 16942 hybrid_clock.cc:648] HybridClock initialized: now 1790266466733303 us; error 27 us; skew 500 ppm
Sep 24 16:14:26 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266466, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:26.736193 16942 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:14:26.737221 16942 webserver.cc:460] Webserver started at http://127.1.253.126:41869/ using document root <none> and password file <none>
I20260924 16:14:26.737828 16942 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:26.737926 16942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:26.742043 16942 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:14:26.744524 16957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:26.745886 16942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:14:26.746136 16942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "76876cc5fc2d4c7fb62f865db014fe49"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "2f01d4ff54eb8b8087459ebb76d5e933"
server_key_iv: "d5dc6962a82f5e34b844ede353e3a743"
server_key_version: "encryptionkey@0"
I20260924 16:14:26.746809 16942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:26.759945 16942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:26.763372 16942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:26.763619 16942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:26.771363 16942 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:45771
I20260924 16:14:26.771663 17010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:45771 every 8 connection(s)
I20260924 16:14:26.772624 16942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:14:26.777163 17011 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:14:26.778514 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 16942
I20260924 16:14:26.779021 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:38779
--local_ip_for_outbound_sockets=127.1.253.65
--tserver_master_addrs=127.1.253.126:45771
--webserver_port=41429
--webserver_interface=127.1.253.65
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:14:26.779273 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Bootstrap starting.
I20260924 16:14:26.797735 17011 log.cc:824] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:26.809686 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:14:26.810122 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49: Bootstrap complete.
I20260924 16:14:26.814561 17011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } }
I20260924 16:14:26.815053 17011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76876cc5fc2d4c7fb62f865db014fe49, State: Initialized, Role: FOLLOWER
I20260924 16:14:26.815620 17011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } }
I20260924 16:14:26.815768 17011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:14:26.815868 17011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:14:26.815989 17011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:14:26.818089 17011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } }
I20260924 16:14:26.818462 17011 leader_election.cc:302] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76876cc5fc2d4c7fb62f865db014fe49; no voters:
I20260924 16:14:26.818801 17011 leader_election.cc:288] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 16:14:26.818941 17015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 16:14:26.819216 17015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [term 3 LEADER]: Becoming Leader. State: Replica: 76876cc5fc2d4c7fb62f865db014fe49, State: Running, Role: LEADER
I20260924 16:14:26.819550 17015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } }
I20260924 16:14:26.820111 17011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:14:26.820567 17015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76876cc5fc2d4c7fb62f865db014fe49. Latest consensus state: current_term: 3 leader_uuid: "76876cc5fc2d4c7fb62f865db014fe49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } } }
I20260924 16:14:26.820677 17015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:26.821287 17016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "76876cc5fc2d4c7fb62f865db014fe49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876cc5fc2d4c7fb62f865db014fe49" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 45771 } } }
I20260924 16:14:26.821463 17016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76876cc5fc2d4c7fb62f865db014fe49 [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:26.822414 17019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:14:26.827795 17019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:14:26.828480 17019 catalog_manager.cc:1259] Loaded cluster ID: 301e9ca6e8634dffba066a28ae2305ef
I20260924 16:14:26.828544 17019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 16:14:26.829607 17019 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f478d86f9ad google::LogMessage::Fail() at ??:0
@ 0x7f478d87288a google::LogMessage::SendToLog() at ??:0
@ 0x7f478d86f4de google::LogMessage::Flush() at ??:0
@ 0x7f478d870a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f47981b2fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f47981b3d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f47981b0579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f47981f4640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f478fac13d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f478fac1c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f478fac33f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f478fab2512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f47910f66db start_thread at ??:0
@ 0x7f478c36d61f clone at ??:0
W20260924 16:14:26.922266 17013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:26.922569 17013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:26.922641 17013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:26.926874 17013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:26.926939 17013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:26.927029 17013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:14:26.932602 17013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:38779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=41429
--tserver_master_addrs=127.1.253.126:45771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:26.934053 17013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:26.935815 17013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:26.948661 17052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:26.952430 17055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:26.952696 17053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:26.953658 17013 server_base.cc:1034] running on GCE node
I20260924 16:14:26.954216 17013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:26.955159 17013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:26.956382 17013 hybrid_clock.cc:648] HybridClock initialized: now 1790266466956356 us; error 36 us; skew 500 ppm
Sep 24 16:14:26 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266466, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:26.960278 17013 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:14:26.961563 17013 webserver.cc:460] Webserver started at http://127.1.253.65:41429/ using document root <none> and password file <none>
I20260924 16:14:26.962412 17013 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:26.962512 17013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:26.968007 17013 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:14:26.971068 17062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:26.972855 17013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:14:26.972997 17013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "2353fc3ef96145278fc62b01dde95288"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "d3922a3f99782138e901eebc45376e0c"
server_key_iv: "af11e315ca356a5a7160c0507cacd9d3"
server_key_version: "encryptionkey@0"
I20260924 16:14:26.973798 17013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:27.001186 17013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:27.005654 17013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:27.005893 17013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:27.006726 17013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:27.008235 17069 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 16:14:27.014460 17013 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:14:27.014554 17013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260924 16:14:27.014631 17013 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:14:27.016995 17013 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:14:27.017062 17013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 16:14:27.017535 17069 tablet_bootstrap.cc:492] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: Bootstrap starting.
I20260924 16:14:27.029745 17013 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:38779
I20260924 16:14:27.032240 17178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:38779 every 8 connection(s)
I20260924 16:14:27.033561 17013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:14:27.041137 17069 log.cc:824] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:27.041934 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 17013
I20260924 16:14:27.042515 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:42023
--local_ip_for_outbound_sockets=127.1.253.66
--tserver_master_addrs=127.1.253.126:45771
--webserver_port=41847
--webserver_interface=127.1.253.66
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:14:27.052049 17069 tablet_bootstrap.cc:492] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:14:27.052568 17069 tablet_bootstrap.cc:492] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: Bootstrap complete.
I20260924 16:14:27.054170 17069 ts_tablet_manager.cc:1397] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: Time spent bootstrapping tablet: real 0.037s user 0.020s sys 0.011s
I20260924 16:14:27.058040 17069 raft_consensus.cc:348] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:27.058656 17069 raft_consensus.cc:729] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2353fc3ef96145278fc62b01dde95288, State: Initialized, Role: FOLLOWER
I20260924 16:14:27.059219 17069 consensus_queue.cc:260] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:27.060336 17069 ts_tablet_manager.cc:1428] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260924 16:14:27.101001 17181 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:14:27.036321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:35701)
0924 16:14:27.037884 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:27.037955 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:27.039251 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:27.099907 (+ 60656us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1293,"thread_start_us":176,"threads_started":1}
W20260924 16:14:27.101673 17179 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:45771 (0 consecutive failures): Network error: Failed to ping master at 127.1.253.126:45771: Client connection negotiation failed: client connection to 127.1.253.126:45771: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 16:14:27.102483 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:27.102169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:52673)
0924 16:14:27.102339 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:27.102416 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 16:14:27.103143 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:27.102917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:56385)
0924 16:14:27.103035 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:27.103087 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 16:14:27.103339 17179 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:14:27.172324 17183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:27.172646 17183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:27.172725 17183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:27.176815 17183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:27.176913 17183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:27.177022 17183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:14:27.182552 17183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:42023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=41847
--tserver_master_addrs=127.1.253.126:45771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:27.183735 17183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:27.184813 17183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:27.194861 17192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:27.194850 17190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:27.194904 17183 server_base.cc:1034] running on GCE node
W20260924 16:14:27.194850 17189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:27.195570 17183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:27.196255 17183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:27.197467 17183 hybrid_clock.cc:648] HybridClock initialized: now 1790266467197427 us; error 50 us; skew 500 ppm
Sep 24 16:14:27 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266467, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:27.200590 17183 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:14:27.201596 17183 webserver.cc:460] Webserver started at http://127.1.253.66:41847/ using document root <none> and password file <none>
I20260924 16:14:27.202219 17183 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:27.202365 17183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:27.206629 17183 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:14:27.209084 17199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:27.210520 17183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:14:27.210654 17183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "bcbc0826a058433092d482b61f0d4b63"
format_stamp: "Formatted at 2026-09-24 16:14:24 on dist-test-slave-6wvm"
server_key: "c7edb68bf1c0010bcb9c8a942dc11073"
server_key_iv: "69d56ec57fced7665a39b654c3c2183a"
server_key_version: "encryptionkey@0"
I20260924 16:14:27.211236 17183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:27.247102 17183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:27.250851 17183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:27.251091 17183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:27.251765 17183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:27.252666 17183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:27.252714 17183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:27.252784 17183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:27.252825 17183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:27.261988 17183 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:42023
I20260924 16:14:27.262014 17313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:42023 every 8 connection(s)
I20260924 16:14:27.263156 17183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:14:27.267189 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:27.265528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:52437)
0924 16:14:27.266252 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:27.266439 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":111,"threads_started":1}
W20260924 16:14:27.267690 17314 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:45771 (0 consecutive failures): Network error: Failed to ping master at 127.1.253.126:45771: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
I20260924 16:14:27.268333 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:27.268110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:59033)
0924 16:14:27.268218 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:27.268268 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 16:14:27.268910 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:27.268726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:45509)
0924 16:14:27.268821 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:27.268863 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 16:14:27.269100 17314 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:14:27.271610 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 17183
I20260924 16:14:27.272044 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:36577
--local_ip_for_outbound_sockets=127.1.253.67
--tserver_master_addrs=127.1.253.126:45771
--webserver_port=41837
--webserver_interface=127.1.253.67
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:27.389086 17317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:27.389370 17317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:27.389424 17317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:27.393395 17317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:27.393482 17317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:27.393543 17317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:14:27.398864 17317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:36577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=41837
--tserver_master_addrs=127.1.253.126:45771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:27.399992 17317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:27.401043 17317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:27.410912 17323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:27.411119 17317 server_base.cc:1034] running on GCE node
W20260924 16:14:27.410974 17325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:27.410912 17322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:27.411643 17317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:27.412322 17317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:27.413486 17317 hybrid_clock.cc:648] HybridClock initialized: now 1790266467413452 us; error 47 us; skew 500 ppm
Sep 24 16:14:27 dist-test-slave-6wvm krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266467, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:27.416554 17317 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:14:27.417531 17317 webserver.cc:460] Webserver started at http://127.1.253.67:41837/ using document root <none> and password file <none>
I20260924 16:14:27.418150 17317 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:27.418241 17317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:27.422390 17317 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 16:14:27.424479 17332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:27.425740 17317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:14:27.425890 17317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "e58776ae6a124a0da6a9c3541adba1c8"
format_stamp: "Formatted at 2026-09-24 16:14:25 on dist-test-slave-6wvm"
server_key: "982841ad1648d7fd12ae7c79fb2f49de"
server_key_iv: "8802374937963b8def61c05ab136dd8b"
server_key_version: "encryptionkey@0"
I20260924 16:14:27.426530 17317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:27.437318 17317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:27.440979 17317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:27.441212 17317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:27.441879 17317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:27.442829 17317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:27.442902 17317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:27.442976 17317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:27.443020 17317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:27.452487 17317 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:36577
I20260924 16:14:27.452513 17447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:36577 every 8 connection(s)
I20260924 16:14:27.453595 17317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:14:27.457489 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:27.455896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:38411)
0924 16:14:27.456614 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:27.456729 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":118,"threads_started":1}
W20260924 16:14:27.458039 17448 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:45771 (0 consecutive failures): Network error: Failed to ping master at 127.1.253.126:45771: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
I20260924 16:14:27.458679 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:27.458466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:36883)
0924 16:14:27.458573 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:27.458623 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 16:14:27.459408 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:27.459122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:35601)
0924 16:14:27.459216 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:27.459261 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 16:14:27.459488 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 17317
W20260924 16:14:27.459645 17448 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:14:28.104589 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:28.104026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:34075)
0924 16:14:28.104383 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:28.104475 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":74,"threads_started":1}
I20260924 16:14:28.270023 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:28.269640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:55639)
0924 16:14:28.269866 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:28.269926 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260924 16:14:28.460659 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:28.460223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:60427)
0924 16:14:28.460485 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:28.460556 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260924 16:14:28.615188 17455 raft_consensus.cc:482] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:14:28.615423 17455 raft_consensus.cc:504] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:28.616643 17455 leader_election.cc:288] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577), bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023)
Sep 24 16:14:28 dist-test-slave-6wvm krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266466, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.66@KRBTEST.COM
Sep 24 16:14:28 dist-test-slave-6wvm krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266466, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.67@KRBTEST.COM
I20260924 16:14:28.635597 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:28.617151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:38193 (local address 127.1.253.66:42023)
0924 16:14:28.617721 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:28.617754 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:28.617823 (+ 69us) server_negotiation.cc:407] Connection header received
0924 16:14:28.618736 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:28.618746 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:28.619248 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:28.619619 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:28.621521 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:28.623203 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:28.625362 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:28.626133 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:28.626359 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:28.629809 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:28.629817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:28.629840 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:28.629882 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:28.632137 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:28.632888 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:28.632895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:28.632902 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:28.633059 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:28.633507 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:28.633510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:28.633513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:28.634069 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:28.634203 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:28.635083 (+ 880us) server_negotiation.cc:299] Negotiation successful
0924 16:14:28.635395 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":86,"threads_started":1}
I20260924 16:14:28.635598 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:28.617173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:44803 (local address 127.1.253.67:36577)
0924 16:14:28.617647 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:28.617679 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:28.617731 (+ 52us) server_negotiation.cc:407] Connection header received
0924 16:14:28.618736 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:28.618746 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:28.619248 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:28.619618 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:28.621520 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:28.623203 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:28.625362 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:28.626133 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:28.626359 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:28.630182 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:28.630188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:28.630203 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:28.630299 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:28.632234 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:28.632888 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:28.632895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:28.632902 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:28.633059 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:28.633502 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:28.633506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:28.633507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:28.634069 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:28.634203 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:28.635083 (+ 880us) server_negotiation.cc:299] Negotiation successful
0924 16:14:28.635395 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":92,"threads_started":1}
I20260924 16:14:28.635617 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:28.617009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:36577 (local address 127.1.253.65:44803)
0924 16:14:28.617557 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:28.617572 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:28.617650 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:28.619742 (+ 2092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:28.619749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:28.620358 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:28.621288 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:28.621296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:28.623401 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:28.623407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:28.625219 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:28.625232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:28.626305 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:28.626309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:28.626581 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:28.627419 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:28.627441 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:28.629921 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:28.632354 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:28.632360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:28.632364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:28.632757 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:28.633174 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:28.633177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:28.633179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:28.633396 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:28.634275 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:28.634283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:28.634689 (+ 406us) client_negotiation.cc:769] Sending connection context
0924 16:14:28.634935 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 16:14:28.635405 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":450,"spinlock_wait_cycles":21376,"thread_start_us":71,"threads_started":1}
I20260924 16:14:28.635617 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:28.616951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:42023 (local address 127.1.253.65:38193)
0924 16:14:28.617327 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:28.617343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:28.617469 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:28.619742 (+ 2273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:28.619749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:28.620553 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:28.621255 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:28.621281 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:28.623410 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:28.623412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:28.625219 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:28.625232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:28.626305 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:28.626309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:28.626582 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:28.627435 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:28.627450 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:28.629529 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:28.632284 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:28.632293 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:28.632311 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:28.632757 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:28.633174 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:28.633177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:28.633179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:28.633396 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:28.634275 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:28.634283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:28.634689 (+ 406us) client_negotiation.cc:769] Sending connection context
0924 16:14:28.634935 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 16:14:28.635459 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":631,"spinlock_wait_cycles":44928,"thread_start_us":93,"threads_started":1}
W20260924 16:14:28.638691 17066 leader_election.cc:341] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577): Not found: Tablet not found: bb2e00834a614bdf943095bb1c94f0da
W20260924 16:14:28.638934 17065 leader_election.cc:341] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023): Not found: Tablet not found: bb2e00834a614bdf943095bb1c94f0da
I20260924 16:14:28.639042 17065 leader_election.cc:302] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2353fc3ef96145278fc62b01dde95288; no voters: bcbc0826a058433092d482b61f0d4b63, e58776ae6a124a0da6a9c3541adba1c8
I20260924 16:14:28.639318 17455 raft_consensus.cc:2726] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:14:29.105878 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:29.105552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:39099)
0924 16:14:29.105719 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:29.105786 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 16:14:29.271382 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:29.270909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:44287)
0924 16:14:29.271191 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:29.271259 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260924 16:14:29.462188 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:29.461681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:49015)
0924 16:14:29.462017 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:29.462087 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":96,"threads_started":1}
I20260924 16:14:30.107245 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:30.106786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:36127)
0924 16:14:30.107078 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:30.107142 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":88,"threads_started":1}
I20260924 16:14:30.272773 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:30.272325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:40545)
0924 16:14:30.272606 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:30.272670 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260924 16:14:30.463457 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:30.462964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:42087)
0924 16:14:30.463272 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:30.463353 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
I20260924 16:14:30.564198 17465 raft_consensus.cc:482] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:14:30.564348 17465 raft_consensus.cc:504] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:30.564914 17465 leader_election.cc:288] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577), bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023)
W20260924 16:14:30.565762 17066 leader_election.cc:341] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577): Not found: Tablet not found: bb2e00834a614bdf943095bb1c94f0da
W20260924 16:14:30.565886 17065 leader_election.cc:341] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023): Not found: Tablet not found: bb2e00834a614bdf943095bb1c94f0da
I20260924 16:14:30.565991 17065 leader_election.cc:302] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2353fc3ef96145278fc62b01dde95288; no voters: bcbc0826a058433092d482b61f0d4b63, e58776ae6a124a0da6a9c3541adba1c8
I20260924 16:14:30.566150 17465 raft_consensus.cc:2726] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:14:31.108422 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.108003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:59509)
0924 16:14:31.108269 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:31.108325 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260924 16:14:31.274123 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.273665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:34973)
0924 16:14:31.273957 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:31.274015 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":76,"threads_started":1}
I20260924 16:14:31.464764 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.464300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:56095)
0924 16:14:31.464588 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:31.464653 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260924 16:14:32.109766 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:32.109319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:35325)
0924 16:14:32.109610 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:32.109667 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":73,"threads_started":1}
I20260924 16:14:32.275344 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:32.274910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:33531)
0924 16:14:32.275184 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:32.275239 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260924 16:14:32.466116 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:32.465668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:42231)
0924 16:14:32.465954 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:32.466016 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260924 16:14:33.111330 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:33.110847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:39839)
0924 16:14:33.111158 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:33.111225 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":103,"threads_started":1}
I20260924 16:14:33.276741 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:33.276238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:39003)
0924 16:14:33.276569 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:33.276635 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":105,"threads_started":1}
I20260924 16:14:33.288597 17474 raft_consensus.cc:482] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:14:33.288754 17474 raft_consensus.cc:504] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:33.289252 17474 leader_election.cc:288] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577), bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023)
W20260924 16:14:33.290256 17066 leader_election.cc:341] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577): Not found: Tablet not found: bb2e00834a614bdf943095bb1c94f0da
W20260924 16:14:33.290357 17065 leader_election.cc:341] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023): Not found: Tablet not found: bb2e00834a614bdf943095bb1c94f0da
I20260924 16:14:33.290439 17065 leader_election.cc:302] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2353fc3ef96145278fc62b01dde95288; no voters: bcbc0826a058433092d482b61f0d4b63, e58776ae6a124a0da6a9c3541adba1c8
I20260924 16:14:33.290630 17474 raft_consensus.cc:2726] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:14:33.467520 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:33.467015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:57655)
0924 16:14:33.467331 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:33.467410 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
I20260924 16:14:34.112931 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:34.112414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:38653)
0924 16:14:34.112723 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:34.112787 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
I20260924 16:14:34.278412 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:34.277720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:55753)
0924 16:14:34.278063 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:34.278131 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":84,"threads_started":1}
I20260924 16:14:34.468851 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:34.468339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:37781)
0924 16:14:34.468657 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:34.468725 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":78,"threads_started":1}
I20260924 16:14:35.114293 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:35.113826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:49181)
0924 16:14:35.114083 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:35.114145 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260924 16:14:35.279907 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:35.279382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:53423)
0924 16:14:35.279687 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:35.279800 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260924 16:14:35.470003 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:35.469549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:42483)
0924 16:14:35.469831 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:35.469893 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
I20260924 16:14:36.115644 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:36.115210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:52317)
0924 16:14:36.115481 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:36.115542 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260924 16:14:36.281180 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:36.280712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:50039)
0924 16:14:36.281015 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:36.281080 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
I20260924 16:14:36.471308 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:36.470863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:54331)
0924 16:14:36.471153 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:36.471208 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260924 16:14:37.117071 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:37.116590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:47073)
0924 16:14:37.116912 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:37.116971 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":90,"threads_started":1}
I20260924 16:14:37.282436 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:37.281933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:34569)
0924 16:14:37.282213 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:37.282337 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260924 16:14:37.472682 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:37.472202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:37809)
0924 16:14:37.472489 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:37.472577 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260924 16:14:37.592587 17488 raft_consensus.cc:482] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:14:37.592761 17488 raft_consensus.cc:504] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e58776ae6a124a0da6a9c3541adba1c8" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36577 } } peers { permanent_uuid: "bcbc0826a058433092d482b61f0d4b63" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42023 } } peers { permanent_uuid: "2353fc3ef96145278fc62b01dde95288" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 38779 } }
I20260924 16:14:37.593178 17488 leader_election.cc:288] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577), bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023)
W20260924 16:14:37.594106 17065 leader_election.cc:341] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bcbc0826a058433092d482b61f0d4b63 (127.1.253.66:42023): Not found: Tablet not found: bb2e00834a614bdf943095bb1c94f0da
W20260924 16:14:37.594223 17066 leader_election.cc:341] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e58776ae6a124a0da6a9c3541adba1c8 (127.1.253.67:36577): Not found: Tablet not found: bb2e00834a614bdf943095bb1c94f0da
I20260924 16:14:37.594310 17066 leader_election.cc:302] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2353fc3ef96145278fc62b01dde95288; no voters: bcbc0826a058433092d482b61f0d4b63, e58776ae6a124a0da6a9c3541adba1c8
I20260924 16:14:37.594483 17488 raft_consensus.cc:2726] T bb2e00834a614bdf943095bb1c94f0da P 2353fc3ef96145278fc62b01dde95288 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:14:38.118543 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:38.118021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:49237)
0924 16:14:38.118357 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:38.118421 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
I20260924 16:14:38.283751 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:38.283292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:58973)
0924 16:14:38.283580 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:38.283647 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260924 16:14:38.474123 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:38.473617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:52805)
0924 16:14:38.473953 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:38.474021 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":83,"threads_started":1}
I20260924 16:14:39.119979 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:39.119522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:55811)
0924 16:14:39.119810 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:39.119875 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
I20260924 16:14:39.285197 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:39.284719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:55951)
0924 16:14:39.285027 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:39.285092 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
I20260924 16:14:39.475620 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:39.475142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:38029)
0924 16:14:39.475443 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:39.475511 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260924 16:14:40.121524 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:40.120938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:51605)
0924 16:14:40.121354 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:40.121423 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":116,"threads_started":1}
I20260924 16:14:40.286646 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:40.286103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:50079)
0924 16:14:40.286482 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:40.286545 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":74,"threads_started":1}
I20260924 16:14:40.477018 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:40.476525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:40197)
0924 16:14:40.476841 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:40.476907 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":79,"threads_started":1}
I20260924 16:14:41.122964 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:41.122501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:43583)
0924 16:14:41.122794 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:41.122860 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260924 16:14:41.287940 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:41.287489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:41241)
0924 16:14:41.287775 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:41.287837 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260924 16:14:41.478467 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:41.477832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.67:41011)
0924 16:14:41.478284 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:41.478363 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":78,"threads_started":1}
I20260924 16:14:42.124397 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:42.123972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.65:49403)
0924 16:14:42.124236 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:42.124295 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260924 16:14:42.289347 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:42.288946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:45771 (local address 127.1.253.66:36551)
0924 16:14:42.289191 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:42.289246 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260924 16:14:42.462843 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 17013
I20260924 16:14:42.470287 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 17183
I20260924 16:14:42.476584 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 17317
2026-09-24T16:14:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19577 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:14:42 dist-test-slave-6wvm krb5kdc[17505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:14:42 dist-test-slave-6wvm krb5kdc[17505](info): set up 2 sockets
Sep 24 16:14:42 dist-test-slave-6wvm krb5kdc[17505](info): commencing operation
krb5kdc: starting...
W20260924 16:14:43.523645 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:14:43 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:14:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:14:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:14:43.654201 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:38839
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:38839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:43.775749 17518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:43.776007 17518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:43.776048 17518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:43.780122 17518 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:14:43.780179 17518 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:14:43.780195 17518 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:14:43.780210 17518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:14:43.780227 17518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:43.780268 17518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:43.780301 17518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:14:43.780323 17518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:14:43.785542 17518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:38839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:38839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:43.786695 17518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:43.787737 17518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:43.797143 17523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:43.797247 17524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:43.797273 17526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:43.797407 17518 server_base.cc:1034] running on GCE node
I20260924 16:14:43.798097 17518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:43.799268 17518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:43.800449 17518 hybrid_clock.cc:648] HybridClock initialized: now 1790266483800429 us; error 31 us; skew 500 ppm
Sep 24 16:14:43 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266483, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:43.811797 17518 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:14:43.812814 17518 webserver.cc:460] Webserver started at http://127.1.253.126:41301/ using document root <none> and password file <none>
I20260924 16:14:43.813390 17518 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:43.813452 17518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:43.813630 17518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:14:43.815779 17518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "aff9e1fcd06d4b0a97d783de156a6b2c"
format_stamp: "Formatted at 2026-09-24 16:14:43 on dist-test-slave-6wvm"
server_key: "e81edce74934470dc77c018def8fa0bd"
server_key_iv: "22db76418ae0f0ebed84adc8ba619057"
server_key_version: "encryptionkey@0"
I20260924 16:14:43.816303 17518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "aff9e1fcd06d4b0a97d783de156a6b2c"
format_stamp: "Formatted at 2026-09-24 16:14:43 on dist-test-slave-6wvm"
server_key: "e81edce74934470dc77c018def8fa0bd"
server_key_iv: "22db76418ae0f0ebed84adc8ba619057"
server_key_version: "encryptionkey@0"
I20260924 16:14:43.820072 17518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:14:43.822609 17533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:43.823896 17518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:14:43.824021 17518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "aff9e1fcd06d4b0a97d783de156a6b2c"
format_stamp: "Formatted at 2026-09-24 16:14:43 on dist-test-slave-6wvm"
server_key: "e81edce74934470dc77c018def8fa0bd"
server_key_iv: "22db76418ae0f0ebed84adc8ba619057"
server_key_version: "encryptionkey@0"
I20260924 16:14:43.824128 17518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:43.846294 17518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:43.849210 17518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:43.849366 17518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:43.856717 17518 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:38839
I20260924 16:14:43.857009 17586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:38839 every 8 connection(s)
I20260924 16:14:43.857826 17518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:14:43.860791 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:43.861234 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 17518
I20260924 16:14:43.861370 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:14:43.861685 2037 external_mini_cluster.cc:1428] Setting key c234f6cd631e6d27ed562ba7c5a58a97
I20260924 16:14:43.868425 17587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Bootstrap starting.
I20260924 16:14:43.871069 17587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Neither blocks nor log segments found. Creating new log.
Sep 24 16:14:43 dist-test-slave-6wvm krb5kdc[17505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:43.872068 17587 log.cc:824] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:43.874462 17587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: No bootstrap required, opened a new log
I20260924 16:14:43.877319 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:43.863247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.1.253.126:38839)
0924 16:14:43.863673 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:43.863692 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:43.863738 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:14:43.864519 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:43.864545 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:43.864992 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:43.865386 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:43.866098 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:43.867654 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:43.868586 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:43.869085 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:43.869259 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:43.872205 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:43.872216 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:43.872243 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:43.872273 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:43.874225 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:43.874779 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:43.874785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:43.874790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:43.874961 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:43.875272 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:43.875274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:43.875275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:43.875703 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:43.875899 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:43.876167 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 16:14:43.876505 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":104,"threads_started":1}
I20260924 16:14:43.878700 17587 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } }
I20260924 16:14:43.878988 17587 raft_consensus.cc:374] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:43.879047 17587 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff9e1fcd06d4b0a97d783de156a6b2c, State: Initialized, Role: FOLLOWER
I20260924 16:14:43.879549 17587 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } }
I20260924 16:14:43.879711 17587 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:14:43.879815 17587 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:14:43.879956 17587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:43.880961 17587 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } }
I20260924 16:14:43.881317 17587 leader_election.cc:302] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff9e1fcd06d4b0a97d783de156a6b2c; no voters:
I20260924 16:14:43.881672 17587 leader_election.cc:288] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:14:43.881861 17592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:14:43.882120 17592 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 1 LEADER]: Becoming Leader. State: Replica: aff9e1fcd06d4b0a97d783de156a6b2c, State: Running, Role: LEADER
I20260924 16:14:43.882901 17592 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } }
I20260924 16:14:43.883092 17587 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:14:43.884716 17594 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff9e1fcd06d4b0a97d783de156a6b2c. Latest consensus state: current_term: 1 leader_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } } }
I20260924 16:14:43.884771 17593 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } } }
I20260924 16:14:43.884865 17594 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:43.884927 17593 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:43.885396 17605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:14:43.889249 17605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:14:43.895592 17605 catalog_manager.cc:1347] Generated new cluster ID: dcbffd456b6a43b5b94669609c9a5432
I20260924 16:14:43.895684 17605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:14:43.925335 17605 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:14:43.926856 17605 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:14:43.940027 17605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Generated new TSK 0
I20260924 16:14:43.941067 17605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:14:43.981091 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38839
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:44.101626 17614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:44.101889 17614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:44.101958 17614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:44.106024 17614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:44.106083 17614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:44.106184 17614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:14:44.111543 17614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:44.112624 17614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:44.113693 17614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:14:44.123575 17614 server_base.cc:1034] running on GCE node
W20260924 16:14:44.123514 17619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:44.123531 17620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:44.123514 17622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:44.124176 17614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:44.124737 17614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:44.125900 17614 hybrid_clock.cc:648] HybridClock initialized: now 1790266484125885 us; error 28 us; skew 500 ppm
Sep 24 16:14:44 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266484, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:44.128854 17614 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:14:44.129742 17614 webserver.cc:460] Webserver started at http://127.1.253.65:36101/ using document root <none> and password file <none>
I20260924 16:14:44.130343 17614 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:44.130411 17614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:44.130589 17614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:14:44.132623 17614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "81b6fe4c1ee2410abd349c07ef551975"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "ad39dae3edc933136396b6b112a8f8d3"
server_key_iv: "1c7ea88f3b9cc59ef802354e519d4a16"
server_key_version: "encryptionkey@0"
I20260924 16:14:44.133147 17614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "81b6fe4c1ee2410abd349c07ef551975"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "ad39dae3edc933136396b6b112a8f8d3"
server_key_iv: "1c7ea88f3b9cc59ef802354e519d4a16"
server_key_version: "encryptionkey@0"
I20260924 16:14:44.136859 17614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:14:44.139359 17629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:44.140616 17614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:14:44.140715 17614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "81b6fe4c1ee2410abd349c07ef551975"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "ad39dae3edc933136396b6b112a8f8d3"
server_key_iv: "1c7ea88f3b9cc59ef802354e519d4a16"
server_key_version: "encryptionkey@0"
I20260924 16:14:44.140803 17614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:44.155200 17614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:44.158632 17614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:44.158845 17614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:44.159590 17614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:44.160478 17614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:44.160527 17614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:44.160604 17614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:44.160636 17614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:44.170046 17614 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:40807
I20260924 16:14:44.170069 17744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:40807 every 8 connection(s)
I20260924 16:14:44.171003 17614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:14:44.178719 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 17614
I20260924 16:14:44.178920 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:14:44.179316 2037 external_mini_cluster.cc:1428] Setting key 8713f0c9c7e3193949bc9c9b3882d2f9
Sep 24 16:14:44 dist-test-slave-6wvm krb5kdc[17505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266484, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:44.186362 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.172920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:40813 (local address 127.1.253.126:38839)
0924 16:14:44.173074 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:44.173078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:44.173976 (+ 898us) server_negotiation.cc:407] Connection header received
0924 16:14:44.175372 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:44.175375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:44.175424 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:44.175499 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:44.176707 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.177240 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.178629 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.178894 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.178999 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.181922 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:44.181931 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:44.181933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:44.181955 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:44.183790 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:44.184386 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:44.184390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:44.184392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:44.184484 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:44.184834 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:44.184838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:44.184840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:44.185036 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:44.185147 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:44.185913 (+ 766us) server_negotiation.cc:299] Negotiation successful
0924 16:14:44.186106 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:14:44.187191 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.173310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:40813)
0924 16:14:44.173805 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:44.173851 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:44.175140 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:44.175675 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:44.175683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:44.176060 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:44.176561 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.176576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.177391 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.177395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.178480 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.178496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.179071 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.179078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.179317 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.180069 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:44.180092 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:44.181749 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:44.183903 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:44.183909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:44.183926 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:44.184283 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:44.184582 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:44.184585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:44.184587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:44.184735 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:44.185148 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:44.185155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:44.185578 (+ 423us) client_negotiation.cc:769] Sending connection context
0924 16:14:44.185824 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 16:14:44.186183 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":106,"threads_started":1}
I20260924 16:14:44.188566 17745 heartbeater.cc:347] Connected to a master server at 127.1.253.126:38839
I20260924 16:14:44.188834 17745 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:44.189352 17745 heartbeater.cc:511] Master 127.1.253.126:38839 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:14:44.190984 17550 ts_manager.cc:194] Registered new tserver with Master: 81b6fe4c1ee2410abd349c07ef551975 (127.1.253.65:40807)
I20260924 16:14:44.192181 17550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:40813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:14:44.208979 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38839
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:44.331306 17750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:44.331560 17750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:44.331642 17750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:44.335805 17750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:44.335876 17750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:44.335937 17750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:14:44.341355 17750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:44.342521 17750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:44.343748 17750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:44.353998 17758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:44.354046 17756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:44.354046 17755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:44.354080 17750 server_base.cc:1034] running on GCE node
I20260924 16:14:44.354804 17750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:44.355445 17750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:44.356632 17750 hybrid_clock.cc:648] HybridClock initialized: now 1790266484356605 us; error 37 us; skew 500 ppm
Sep 24 16:14:44 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266484, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:44.359558 17750 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:14:44.360520 17750 webserver.cc:460] Webserver started at http://127.1.253.66:39257/ using document root <none> and password file <none>
I20260924 16:14:44.361151 17750 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:44.361243 17750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:44.361464 17750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:14:44.363667 17750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "a2a0c07dbed84961aa6580b1b662b221"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "0d132c0bcdf69d0bdb3a6acb4230edaf"
server_key_iv: "a1983456f3946248b41c96f6f4d8e05c"
server_key_version: "encryptionkey@0"
I20260924 16:14:44.364244 17750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "a2a0c07dbed84961aa6580b1b662b221"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "0d132c0bcdf69d0bdb3a6acb4230edaf"
server_key_iv: "a1983456f3946248b41c96f6f4d8e05c"
server_key_version: "encryptionkey@0"
I20260924 16:14:44.368166 17750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:14:44.370716 17765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:44.372016 17750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:14:44.372176 17750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "a2a0c07dbed84961aa6580b1b662b221"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "0d132c0bcdf69d0bdb3a6acb4230edaf"
server_key_iv: "a1983456f3946248b41c96f6f4d8e05c"
server_key_version: "encryptionkey@0"
I20260924 16:14:44.372298 17750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:44.386314 17750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:44.389314 17750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:44.389531 17750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:44.390157 17750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:44.391088 17750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:44.391160 17750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:44.391240 17750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:44.391292 17750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:44.400385 17750 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:46797
I20260924 16:14:44.400403 17878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:46797 every 8 connection(s)
I20260924 16:14:44.401376 17750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:14:44.406561 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 17750
I20260924 16:14:44.406682 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:14:44.407003 2037 external_mini_cluster.cc:1428] Setting key 27390621e7dcb721f11040e1681ac785
Sep 24 16:14:44 dist-test-slave-6wvm krb5kdc[17505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266484, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:44.416980 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.403347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:59355 (local address 127.1.253.126:38839)
0924 16:14:44.403489 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:44.403493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:44.404817 (+ 1324us) server_negotiation.cc:407] Connection header received
0924 16:14:44.405742 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:44.405745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:44.405798 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:44.405892 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:44.407428 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.408079 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.409379 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.409615 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.409729 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.412694 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:44.412700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:44.412707 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:44.412745 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:44.414637 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:44.415214 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:44.415221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:44.415225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:44.415334 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:44.415700 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:44.415704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:44.415706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:44.415962 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:44.416083 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:44.416611 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 16:14:44.416768 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:14:44.417773 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.403748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:59355)
0924 16:14:44.404640 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:44.404687 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:44.405501 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:44.406091 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:44.406099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:44.406575 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:44.407253 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.407277 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.408222 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.408225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.409242 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.409253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.409790 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.409794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.409968 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.410681 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:44.410710 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:44.412495 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:44.414782 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:44.414787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:44.414803 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:44.415105 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:44.415438 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:44.415440 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:44.415442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:44.415597 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:44.416077 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:44.416083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:44.416339 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 16:14:44.416509 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 16:14:44.416793 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":542,"threads_started":1}
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:14:44.419149 17879 heartbeater.cc:347] Connected to a master server at 127.1.253.126:38839
I20260924 16:14:44.419384 17879 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:44.419876 17879 heartbeater.cc:511] Master 127.1.253.126:38839 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
I20260924 16:14:44.421165 17550 ts_manager.cc:194] Registered new tserver with Master: a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797)
I20260924 16:14:44.422060 17550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:59355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:14:44.438169 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38839
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:44.554942 17884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:44.555246 17884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:44.555320 17884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:44.559510 17884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:44.559592 17884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:44.559721 17884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:14:44.565502 17884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:38839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:14:44.566757 17884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:44.567848 17884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:44.579988 17890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:44.580000 17891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:44.580114 17893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:44.580507 17884 server_base.cc:1034] running on GCE node
I20260924 16:14:44.581079 17884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:44.581964 17884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:44.583163 17884 hybrid_clock.cc:648] HybridClock initialized: now 1790266484583113 us; error 59 us; skew 500 ppm
Sep 24 16:14:44 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266484, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:44.586946 17884 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:14:44.588006 17884 webserver.cc:460] Webserver started at http://127.1.253.67:40059/ using document root <none> and password file <none>
I20260924 16:14:44.588616 17884 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:44.588681 17884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:44.588910 17884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:14:44.591174 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "2a19522336bb40afb12fa58f743d9367"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "bd377763b810f26d4d28dd9206bbb184"
server_key_iv: "be060443c8f5f5b1d9c2b5f136613dca"
server_key_version: "encryptionkey@0"
I20260924 16:14:44.591794 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "2a19522336bb40afb12fa58f743d9367"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "bd377763b810f26d4d28dd9206bbb184"
server_key_iv: "be060443c8f5f5b1d9c2b5f136613dca"
server_key_version: "encryptionkey@0"
I20260924 16:14:44.595956 17884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:14:44.598804 17900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:44.600342 17884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:14:44.600479 17884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "2a19522336bb40afb12fa58f743d9367"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "bd377763b810f26d4d28dd9206bbb184"
server_key_iv: "be060443c8f5f5b1d9c2b5f136613dca"
server_key_version: "encryptionkey@0"
I20260924 16:14:44.600620 17884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:44.625602 17884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:44.629357 17884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:44.629580 17884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:44.630203 17884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:44.631093 17884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:44.631168 17884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:44.631244 17884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:44.631289 17884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:44.640055 17884 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:37625
I20260924 16:14:44.640131 18016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:37625 every 8 connection(s)
I20260924 16:14:44.641094 17884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:14:44.645613 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 17884
I20260924 16:14:44.645747 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:14:44.646063 2037 external_mini_cluster.cc:1428] Setting key 971d5d49923ad8476702f7b82c919bae
Sep 24 16:14:44 dist-test-slave-6wvm krb5kdc[17505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266484, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:14:44.658521 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.643417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:33783 (local address 127.1.253.126:38839)
0924 16:14:44.643576 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:44.643581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:44.644692 (+ 1111us) server_negotiation.cc:407] Connection header received
0924 16:14:44.646780 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:44.646785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:44.646841 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:44.646920 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:44.648736 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.649265 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.651328 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.651557 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.651639 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.654773 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:44.654782 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:44.654784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:44.654811 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:44.656305 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:44.656865 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:44.656868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:44.656870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:44.656954 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:44.657330 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:44.657334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:44.657335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:44.657541 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:44.657650 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:44.658188 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 16:14:44.658343 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:14:44.659657 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.643874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:33783)
0924 16:14:44.644490 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:44.644562 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:44.646524 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:44.647130 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:44.647143 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:44.647741 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:44.648579 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.648602 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.649409 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.649416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.651178 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.651193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.651750 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.651762 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.652024 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.652689 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:44.652717 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:44.654588 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:44.656435 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:44.656441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:44.656460 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:44.656748 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:44.657065 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:44.657068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:44.657070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:44.657225 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:44.657649 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:44.657658 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:44.657930 (+ 272us) client_negotiation.cc:769] Sending connection context
0924 16:14:44.658129 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 16:14:44.658570 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":114,"threads_started":1}
I20260924 16:14:44.661077 18017 heartbeater.cc:347] Connected to a master server at 127.1.253.126:38839
I20260924 16:14:44.661345 18017 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:44.661872 18017 heartbeater.cc:511] Master 127.1.253.126:38839 requested a full tablet report, sending...
I20260924 16:14:44.663045 17549 ts_manager.cc:194] Registered new tserver with Master: 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625)
I20260924 16:14:44.663654 17549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:33783
I20260924 16:14:44.672957 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:14:44.684514 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.675109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.1.253.126:38839)
0924 16:14:44.675264 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:44.675269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:44.675552 (+ 283us) server_negotiation.cc:407] Connection header received
0924 16:14:44.675735 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:44.675737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:44.675780 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:44.675844 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:44.676904 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.677422 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.678924 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.679141 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.679220 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.681113 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:44.681117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:44.681119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:44.681144 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:44.682678 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:44.683203 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:44.683206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:44.683207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:44.683283 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:44.683594 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:44.683596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:44.683597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:44.683802 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:44.683921 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:44.684255 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 16:14:44.684360 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:14:44.688201 17549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:14:44.694227 17549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:14:44.711700 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.704741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40807 (local address 127.0.0.1:39960)
0924 16:14:44.705202 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:44.705263 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:44.705543 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:44.706585 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:44.706593 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:44.706626 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:44.706785 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.706793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.708382 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.708385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.709254 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.709263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.710875 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.710881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.711091 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.711150 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 16:14:44.711279 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 16:14:44.711457 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":30,"thread_start_us":125,"threads_started":1}
I20260924 16:14:44.712376 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.706037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:46797 (local address 127.0.0.1:52600)
0924 16:14:44.706865 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:44.706882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:44.706976 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:44.707549 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:44.707558 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:44.707577 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:44.707962 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.707973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.709631 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.709633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.710832 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.710841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.711970 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.711975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.712091 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.712115 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:14:44.712162 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 16:14:44.712223 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":27,"thread_start_us":107,"threads_started":1}
I20260924 16:14:44.712471 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.704936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.1.253.65:40807)
0924 16:14:44.705935 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:44.705941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:44.705986 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:14:44.706074 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:44.706079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:44.706345 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:44.706971 (+ 626us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:44.707163 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.708254 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.709397 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.710468 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.710599 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.710802 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:44.712102 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0924 16:14:44.712252 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":117,"threads_started":1}
I20260924 16:14:44.713153 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.706223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.1.253.66:46797)
0924 16:14:44.707052 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:44.707067 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:44.707093 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:14:44.707172 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:44.707180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:44.707397 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:44.708121 (+ 724us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:44.708284 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.709483 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.710965 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.711823 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.712680 (+ 857us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.712846 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:44.712901 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 16:14:44.712998 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":141,"threads_started":1}
I20260924 16:14:44.714550 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.706370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:37625 (local address 127.0.0.1:43026)
0924 16:14:44.707331 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:44.707343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:44.707430 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:44.708081 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:44.708085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:44.708104 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:44.708207 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.708212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.709618 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.709622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.712275 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.712284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.714057 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.714060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.714172 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.714195 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:14:44.714334 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 16:14:44.714410 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":57,"thread_start_us":94,"threads_started":1}
I20260924 16:14:44.714607 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.706542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.1.253.67:37625)
0924 16:14:44.706936 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:44.706949 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:44.707540 (+ 591us) server_negotiation.cc:407] Connection header received
0924 16:14:44.707623 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:44.707628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:44.707839 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:44.708096 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:44.708307 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.709447 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.712410 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.713737 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.713840 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.713996 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:44.714370 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 16:14:44.714468 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":149,"threads_started":1}
I20260924 16:14:44.715896 17677 tablet_service.cc:1467] Processing CreateTablet for tablet ea603e5f59be41a8ad5ff77be07f164c (DEFAULT_TABLE table=test-table [id=a1cde93e59de4e548650e34c2d0f38e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:44.716002 17813 tablet_service.cc:1467] Processing CreateTablet for tablet ea603e5f59be41a8ad5ff77be07f164c (DEFAULT_TABLE table=test-table [id=a1cde93e59de4e548650e34c2d0f38e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:44.717104 17677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea603e5f59be41a8ad5ff77be07f164c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:44.717104 17813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea603e5f59be41a8ad5ff77be07f164c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:44.717144 17948 tablet_service.cc:1467] Processing CreateTablet for tablet ea603e5f59be41a8ad5ff77be07f164c (DEFAULT_TABLE table=test-table [id=a1cde93e59de4e548650e34c2d0f38e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:44.718371 17948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea603e5f59be41a8ad5ff77be07f164c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:44.726312 18035 tablet_bootstrap.cc:492] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Bootstrap starting.
I20260924 16:14:44.726928 18037 tablet_bootstrap.cc:492] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367: Bootstrap starting.
I20260924 16:14:44.727305 18036 tablet_bootstrap.cc:492] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221: Bootstrap starting.
I20260924 16:14:44.729914 18037 tablet_bootstrap.cc:654] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:44.730366 18035 tablet_bootstrap.cc:654] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:44.730919 18037 log.cc:824] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:44.731178 18035 log.cc:824] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:44.733108 18036 tablet_bootstrap.cc:654] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:44.733134 18037 tablet_bootstrap.cc:492] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367: No bootstrap required, opened a new log
I20260924 16:14:44.733343 18037 ts_tablet_manager.cc:1397] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 16:14:44.733955 18036 log.cc:824] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:44.735929 18035 tablet_bootstrap.cc:492] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: No bootstrap required, opened a new log
I20260924 16:14:44.736125 18035 ts_tablet_manager.cc:1397] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20260924 16:14:44.737342 18037 raft_consensus.cc:348] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } }
I20260924 16:14:44.737673 18037 raft_consensus.cc:374] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:44.737742 18037 raft_consensus.cc:729] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a19522336bb40afb12fa58f743d9367, State: Initialized, Role: FOLLOWER
I20260924 16:14:44.738284 18037 consensus_queue.cc:260] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } }
I20260924 16:14:44.739310 18037 ts_tablet_manager.cc:1428] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 16:14:44.739692 18017 heartbeater.cc:503] Master 127.1.253.126:38839 was elected leader, sending a full tablet report...
I20260924 16:14:44.740607 18035 raft_consensus.cc:348] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } }
I20260924 16:14:44.740890 18035 raft_consensus.cc:374] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:44.740942 18035 raft_consensus.cc:729] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b6fe4c1ee2410abd349c07ef551975, State: Initialized, Role: FOLLOWER
I20260924 16:14:44.741400 18035 consensus_queue.cc:260] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } }
I20260924 16:14:44.743000 18035 ts_tablet_manager.cc:1428] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260924 16:14:44.743333 17745 heartbeater.cc:503] Master 127.1.253.126:38839 was elected leader, sending a full tablet report...
I20260924 16:14:44.744910 18036 tablet_bootstrap.cc:492] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221: No bootstrap required, opened a new log
I20260924 16:14:44.745183 18036 ts_tablet_manager.cc:1397] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260924 16:14:44.750733 18036 raft_consensus.cc:348] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } }
I20260924 16:14:44.751080 18036 raft_consensus.cc:374] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:44.751154 18036 raft_consensus.cc:729] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a0c07dbed84961aa6580b1b662b221, State: Initialized, Role: FOLLOWER
I20260924 16:14:44.751667 18036 consensus_queue.cc:260] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } }
I20260924 16:14:44.752382 17879 heartbeater.cc:503] Master 127.1.253.126:38839 was elected leader, sending a full tablet report...
I20260924 16:14:44.752872 18036 ts_tablet_manager.cc:1428] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260924 16:14:44.819026 18043 raft_consensus.cc:482] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:14:44.819247 18043 raft_consensus.cc:504] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } }
I20260924 16:14:44.820590 18043 leader_election.cc:288] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797), 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625)
I20260924 16:14:44.825278 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.820994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:55681 (local address 127.1.253.66:46797)
0924 16:14:44.821135 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:44.821140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:44.821331 (+ 191us) server_negotiation.cc:407] Connection header received
0924 16:14:44.821526 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:44.821529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:44.821575 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:44.821661 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:44.821974 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.822549 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.824002 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.824588 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.824668 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.824710 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:44.825076 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 16:14:44.825149 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:14:44.825278 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.820993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:36993 (local address 127.1.253.67:37625)
0924 16:14:44.821234 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:44.821239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:44.821368 (+ 129us) server_negotiation.cc:407] Connection header received
0924 16:14:44.821531 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:44.821534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:44.821584 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:44.821673 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:44.821986 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.822554 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.824051 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.824646 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.824775 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.824825 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:44.825075 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 16:14:44.825149 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260924 16:14:44.825310 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.820809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:37625 (local address 127.1.253.65:36993)
0924 16:14:44.821262 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:44.821275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:44.821416 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:44.821685 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:44.821688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:44.821718 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:44.821874 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.821881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.822665 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.822670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.823874 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.823889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.824773 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.824777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.824938 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.824961 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:14:44.825059 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 16:14:44.825139 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":143,"spinlock_wait_cycles":3200,"thread_start_us":133,"threads_started":1}
I20260924 16:14:44.825304 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.820809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:46797 (local address 127.1.253.65:55681)
0924 16:14:44.821216 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:44.821230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:44.821398 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:44.821682 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:44.821685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:44.821711 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:44.821858 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.821865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.822657 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.822662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.823835 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:44.823851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.824790 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.824793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:44.824919 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.824962 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 16:14:44.825046 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 16:14:44.825132 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":142,"spinlock_wait_cycles":1920,"thread_start_us":166,"threads_started":1}
I20260924 16:14:44.826001 17968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea603e5f59be41a8ad5ff77be07f164c" candidate_uuid: "81b6fe4c1ee2410abd349c07ef551975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a19522336bb40afb12fa58f743d9367" is_pre_election: true
I20260924 16:14:44.826184 17833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea603e5f59be41a8ad5ff77be07f164c" candidate_uuid: "81b6fe4c1ee2410abd349c07ef551975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a0c07dbed84961aa6580b1b662b221" is_pre_election: true
I20260924 16:14:44.826385 17968 raft_consensus.cc:2454] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b6fe4c1ee2410abd349c07ef551975 in term 0.
I20260924 16:14:44.826568 17833 raft_consensus.cc:2454] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b6fe4c1ee2410abd349c07ef551975 in term 0.
I20260924 16:14:44.826907 17633 leader_election.cc:302] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a19522336bb40afb12fa58f743d9367, 81b6fe4c1ee2410abd349c07ef551975; no voters:
I20260924 16:14:44.827271 18043 raft_consensus.cc:2781] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:14:44.827364 18043 raft_consensus.cc:482] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:14:44.827423 18043 raft_consensus.cc:3037] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:44.828629 18043 raft_consensus.cc:504] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } }
I20260924 16:14:44.829018 18043 leader_election.cc:288] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 1 election: Requested vote from peers a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797), 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625)
I20260924 16:14:44.829397 17968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea603e5f59be41a8ad5ff77be07f164c" candidate_uuid: "81b6fe4c1ee2410abd349c07ef551975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a19522336bb40afb12fa58f743d9367"
I20260924 16:14:44.829411 17833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea603e5f59be41a8ad5ff77be07f164c" candidate_uuid: "81b6fe4c1ee2410abd349c07ef551975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a0c07dbed84961aa6580b1b662b221"
I20260924 16:14:44.829542 17833 raft_consensus.cc:3037] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:44.829546 17968 raft_consensus.cc:3037] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:44.830678 17833 raft_consensus.cc:2454] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b6fe4c1ee2410abd349c07ef551975 in term 1.
I20260924 16:14:44.830698 17968 raft_consensus.cc:2454] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b6fe4c1ee2410abd349c07ef551975 in term 1.
I20260924 16:14:44.831022 17630 leader_election.cc:302] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81b6fe4c1ee2410abd349c07ef551975, a2a0c07dbed84961aa6580b1b662b221; no voters:
I20260924 16:14:44.831245 18043 raft_consensus.cc:2781] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:14:44.831466 18043 raft_consensus.cc:686] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 1 LEADER]: Becoming Leader. State: Replica: 81b6fe4c1ee2410abd349c07ef551975, State: Running, Role: LEADER
I20260924 16:14:44.831818 18043 consensus_queue.cc:237] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } }
I20260924 16:14:44.835219 17549 catalog_manager.cc:5495] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81b6fe4c1ee2410abd349c07ef551975 (127.1.253.65). New cstate: current_term: 1 leader_uuid: "81b6fe4c1ee2410abd349c07ef551975" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } health_report { overall_health: UNKNOWN } } }
I20260924 16:14:44.866714 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:44.862562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.1.253.65:40807)
0924 16:14:44.862694 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:44.862698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:44.862711 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:14:44.862860 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:44.862863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:44.862921 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:44.863003 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:14:44.863282 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.863849 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.865106 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:44.865329 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:44.865421 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:44.865773 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:14:44.866194 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:14:44.866400 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:44.866464 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 16:14:44.866523 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:14:44.873456 17833 raft_consensus.cc:1261] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221 [term 1 FOLLOWER]: Refusing update from remote peer 81b6fe4c1ee2410abd349c07ef551975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:14:44.873517 17968 raft_consensus.cc:1261] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367 [term 1 FOLLOWER]: Refusing update from remote peer 81b6fe4c1ee2410abd349c07ef551975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:14:44.874396 18043 consensus_queue.cc:1035] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:14:44.874635 18047 consensus_queue.cc:1035] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:14:44.881639 18050 mvcc.cc:205] Tried to move back new op lower bound from 7332931522033815552 to 7332931521873887232. Current Snapshot: MvccSnapshot[applied={T|T < 7332931522033815552}]
I20260924 16:14:44.886736 18052 mvcc.cc:205] Tried to move back new op lower bound from 7332931522033815552 to 7332931521873887232. Current Snapshot: MvccSnapshot[applied={T|T < 7332931522033815552}]
I20260924 16:14:44.894589 17548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:14:44.894809 17548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:14:44.897605 17548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Sending DeleteTablet for 3 replicas of tablet ea603e5f59be41a8ad5ff77be07f164c
I20260924 16:14:44.898510 17813 tablet_service.cc:1514] Processing DeleteTablet for tablet ea603e5f59be41a8ad5ff77be07f164c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:52600
I20260924 16:14:44.898592 17948 tablet_service.cc:1514] Processing DeleteTablet for tablet ea603e5f59be41a8ad5ff77be07f164c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:43026
I20260924 16:14:44.898607 17677 tablet_service.cc:1514] Processing DeleteTablet for tablet ea603e5f59be41a8ad5ff77be07f164c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:39960
I20260924 16:14:44.899083 18060 tablet_replica.cc:330] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367: stopping tablet replica
I20260924 16:14:44.899269 18062 tablet_replica.cc:330] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: stopping tablet replica
I20260924 16:14:44.899399 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 17614
I20260924 16:14:44.899451 18060 raft_consensus.cc:2229] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:14:44.899693 18062 raft_consensus.cc:2229] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:14:44.899827 18060 raft_consensus.cc:2258] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:44.901021 18061 tablet_replica.cc:330] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221: stopping tablet replica
I20260924 16:14:44.901306 18061 raft_consensus.cc:2229] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:14:44.901629 18061 raft_consensus.cc:2258] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:44.902902 18060 ts_tablet_manager.cc:1905] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:14:44.903110 18061 ts_tablet_manager.cc:1905] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:14:44.905779 18060 ts_tablet_manager.cc:1918] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:14:44.905779 18061 ts_tablet_manager.cc:1918] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:14:44.905895 18061 log.cc:1196] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/ea603e5f59be41a8ad5ff77be07f164c
I20260924 16:14:44.905897 18060 log.cc:1196] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/ea603e5f59be41a8ad5ff77be07f164c
I20260924 16:14:44.906229 18061 ts_tablet_manager.cc:1939] T ea603e5f59be41a8ad5ff77be07f164c P a2a0c07dbed84961aa6580b1b662b221: Deleting consensus metadata
I20260924 16:14:44.906229 18060 ts_tablet_manager.cc:1939] T ea603e5f59be41a8ad5ff77be07f164c P 2a19522336bb40afb12fa58f743d9367: Deleting consensus metadata
W20260924 16:14:44.907282 17537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:40807 (error 108)
I20260924 16:14:44.907817 17537 catalog_manager.cc:4841] TS 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625): tablet ea603e5f59be41a8ad5ff77be07f164c (table test-table [id=a1cde93e59de4e548650e34c2d0f38e3]) successfully deleted
I20260924 16:14:44.908074 17534 catalog_manager.cc:4841] TS a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797): tablet ea603e5f59be41a8ad5ff77be07f164c (table test-table [id=a1cde93e59de4e548650e34c2d0f38e3]) successfully deleted
W20260924 16:14:44.908703 18031 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:14:44.908184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40807 (local address 127.0.0.1:39974)
0924 16:14:44.908346 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:44.908358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:44.908428 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:44.908616 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:40807: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 16:14:44.908877 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 17750
W20260924 16:14:44.908960 17537 catalog_manager.cc:4584] TS 81b6fe4c1ee2410abd349c07ef551975 (127.1.253.65:40807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea603e5f59be41a8ad5ff77be07f164c: Network error: Client connection negotiation failed: client connection to 127.1.253.65:40807: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 16:14:44.915359 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 17884
I20260924 16:14:44.921774 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 17518
I20260924 16:15:04.929034 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:38839
--webserver_interface=127.1.253.126
--webserver_port=41301
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:38839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:05.043905 18065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:05.044198 18065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:05.044238 18065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:05.048303 18065 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:15:05.048358 18065 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:15:05.048375 18065 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:15:05.048388 18065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:05.048406 18065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:05.048424 18065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:05.048493 18065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:05.048514 18065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:05.055193 18065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:38839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:38839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=41301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:05.056336 18065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:05.057433 18065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:05.071859 18072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:05.071872 18075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:05.072105 18065 server_base.cc:1034] running on GCE node
W20260924 16:15:05.071887 18073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:05.072839 18065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:05.074108 18065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:05.075297 18065 hybrid_clock.cc:648] HybridClock initialized: now 1790266505075260 us; error 56 us; skew 500 ppm
Sep 24 16:15:05 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266505, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:05.087394 18065 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:15:05.088414 18065 webserver.cc:460] Webserver started at http://127.1.253.126:41301/ using document root <none> and password file <none>
I20260924 16:15:05.089044 18065 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:05.089144 18065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:05.093297 18065 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:15:05.095752 18082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:05.097079 18065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:15:05.097218 18065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "aff9e1fcd06d4b0a97d783de156a6b2c"
format_stamp: "Formatted at 2026-09-24 16:14:43 on dist-test-slave-6wvm"
server_key: "e81edce74934470dc77c018def8fa0bd"
server_key_iv: "22db76418ae0f0ebed84adc8ba619057"
server_key_version: "encryptionkey@0"
I20260924 16:15:05.097837 18065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:05.124611 18065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:05.127835 18065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:05.128062 18065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:05.135614 18065 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:38839
I20260924 16:15:05.135938 18138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:38839 every 8 connection(s)
I20260924 16:15:05.136816 18065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:05.141398 18139 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:15:05.143246 18139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Bootstrap starting.
I20260924 16:15:05.146618 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 18065
I20260924 16:15:05.147080 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:40807
--local_ip_for_outbound_sockets=127.1.253.65
--tserver_master_addrs=127.1.253.126:38839
--webserver_port=36101
--webserver_interface=127.1.253.65
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:15:05.161239 18139 log.cc:824] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:05.170677 18139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:15:05.171105 18139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Bootstrap complete.
I20260924 16:15:05.175720 18139 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } }
I20260924 16:15:05.176146 18139 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff9e1fcd06d4b0a97d783de156a6b2c, State: Initialized, Role: FOLLOWER
I20260924 16:15:05.176651 18139 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } }
I20260924 16:15:05.176801 18139 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:05.176908 18139 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:05.177031 18139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:15:05.179059 18139 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } }
I20260924 16:15:05.179409 18139 leader_election.cc:302] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff9e1fcd06d4b0a97d783de156a6b2c; no voters:
I20260924 16:15:05.179773 18139 leader_election.cc:288] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:15:05.179862 18143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:15:05.180102 18143 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 2 LEADER]: Becoming Leader. State: Replica: aff9e1fcd06d4b0a97d783de156a6b2c, State: Running, Role: LEADER
I20260924 16:15:05.180419 18143 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } }
I20260924 16:15:05.181174 18139 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:05.181799 18144 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } } }
I20260924 16:15:05.181924 18144 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:05.182267 18145 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff9e1fcd06d4b0a97d783de156a6b2c. Latest consensus state: current_term: 2 leader_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } } }
I20260924 16:15:05.182369 18145 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:05.182746 18149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:05.187721 18149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:05.188411 18149 catalog_manager.cc:1259] Loaded cluster ID: dcbffd456b6a43b5b94669609c9a5432
I20260924 16:15:05.188473 18149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:05.191068 18149 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:05.191881 18149 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Loaded TSK: 0
I20260924 16:15:05.206910 18149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Generated new TSK 1
I20260924 16:15:05.207197 18149 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:15:05.266667 18141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:05.266968 18141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:05.267041 18141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:05.271229 18141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:05.271318 18141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:05.271407 18141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:15:05.276997 18141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:40807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=36101
--tserver_master_addrs=127.1.253.126:38839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:05.278060 18141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:05.279160 18141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:05.289104 18171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:05.289143 18168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:05.289143 18169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:05.289712 18141 server_base.cc:1034] running on GCE node
I20260924 16:15:05.290120 18141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:05.290815 18141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:05.291997 18141 hybrid_clock.cc:648] HybridClock initialized: now 1790266505291968 us; error 45 us; skew 500 ppm
Sep 24 16:15:05 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266505, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:05.294811 18141 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:15:05.295805 18141 webserver.cc:460] Webserver started at http://127.1.253.65:36101/ using document root <none> and password file <none>
I20260924 16:15:05.296370 18141 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:05.296437 18141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:05.300417 18141 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:05.302825 18178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:05.304141 18141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:05.304239 18141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "81b6fe4c1ee2410abd349c07ef551975"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "ad39dae3edc933136396b6b112a8f8d3"
server_key_iv: "1c7ea88f3b9cc59ef802354e519d4a16"
server_key_version: "encryptionkey@0"
I20260924 16:15:05.304813 18141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:05.330781 18141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:05.333784 18141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:05.333999 18141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:05.334669 18141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:05.335775 18185 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 16:15:05.340164 18141 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:15:05.340269 18141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 16:15:05.340358 18141 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:15:05.342185 18141 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:15:05.342293 18141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260924 16:15:05.342648 18185 tablet_bootstrap.cc:492] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Bootstrap starting.
I20260924 16:15:05.353329 18141 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:40807
I20260924 16:15:05.353364 18293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:40807 every 8 connection(s)
I20260924 16:15:05.354425 18141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:05.355439 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 18141
I20260924 16:15:05.355885 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:46797
--local_ip_for_outbound_sockets=127.1.253.66
--tserver_master_addrs=127.1.253.126:38839
--webserver_port=39257
--webserver_interface=127.1.253.66
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:15:05.363299 18185 log.cc:824] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:15:05 dist-test-slave-6wvm krb5kdc[17505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266505, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:15:05.373425 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.356985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:44475)
0924 16:15:05.358095 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:05.358159 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:05.359556 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:05.360888 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:05.360897 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:05.361360 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:05.361852 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.361867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.363360 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.363363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.364600 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.364612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.365405 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.365409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.365602 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.366312 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:05.366331 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:05.367889 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:05.370113 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:05.370119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:05.370133 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:05.370491 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:05.370866 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:05.370870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:05.370872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:05.371018 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:05.371638 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:05.371645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:05.371903 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 16:15:05.372096 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 16:15:05.372456 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":139,"threads_started":1}
I20260924 16:15:05.373476 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.357046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:44475 (local address 127.1.253.126:38839)
0924 16:15:05.357694 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:05.357723 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:05.358355 (+ 632us) server_negotiation.cc:407] Connection header received
0924 16:15:05.360048 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:05.360094 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:05.360441 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:05.360827 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:05.362050 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.363155 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.364738 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.365239 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.365437 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.368122 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:05.368129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:05.368144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:05.368176 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:05.369974 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:05.370630 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:05.370636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:05.370641 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:05.370760 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:05.371139 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:05.371142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:05.371143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:05.371499 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:05.371669 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:05.372233 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 16:15:05.372506 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":158,"threads_started":1}
I20260924 16:15:05.374400 18185 tablet_bootstrap.cc:492] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:15:05.374930 18185 tablet_bootstrap.cc:492] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Bootstrap complete.
I20260924 16:15:05.376407 18294 heartbeater.cc:347] Connected to a master server at 127.1.253.126:38839
I20260924 16:15:05.376417 18185 ts_tablet_manager.cc:1397] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.013s
I20260924 16:15:05.376720 18294 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:05.377266 18294 heartbeater.cc:511] Master 127.1.253.126:38839 requested a full tablet report, sending...
I20260924 16:15:05.380013 18101 ts_manager.cc:194] Registered new tserver with Master: 81b6fe4c1ee2410abd349c07ef551975 (127.1.253.65:40807)
I20260924 16:15:05.379905 18185 raft_consensus.cc:348] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } }
I20260924 16:15:05.380506 18185 raft_consensus.cc:729] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b6fe4c1ee2410abd349c07ef551975, State: Initialized, Role: FOLLOWER
I20260924 16:15:05.381114 18185 consensus_queue.cc:260] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } }
I20260924 16:15:05.382447 18185 ts_tablet_manager.cc:1428] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:15:05.384473 18101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:44475
I20260924 16:15:05.387259 18294 heartbeater.cc:503] Master 127.1.253.126:38839 was elected leader, sending a full tablet report...
Sep 24 16:15:05 dist-test-slave-6wvm krb5kdc[17505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266505, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.65@KRBTEST.COM
I20260924 16:15:05.394833 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.383648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40807 (local address 127.0.0.1:58012)
0924 16:15:05.384109 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:05.384159 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:05.384286 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:05.385056 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:05.385064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:05.385602 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:05.385769 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.385783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.387129 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.387133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.388093 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.388108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.388869 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.388873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.388994 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.389609 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:05.389621 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:05.391035 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:05.392935 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:05.392946 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:05.392951 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:05.393197 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:05.393519 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:05.393523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:05.393525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:05.393621 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:05.394079 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:05.394086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:05.394216 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 16:15:05.394417 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 16:15:05.394618 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":133,"threads_started":1}
I20260924 16:15:05.394832 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.383694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.1.253.65:40807)
0924 16:15:05.384555 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:05.384569 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:05.384595 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:15:05.384666 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:05.384671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:05.384873 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:05.385018 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:05.385922 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.386988 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.388261 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.388763 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.388870 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.391177 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:05.391187 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:05.391200 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:05.391233 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:05.392816 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:05.393326 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:05.393332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:05.393334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:05.393422 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:05.393713 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:05.393717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:05.393719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:05.393956 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:05.394112 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:05.394503 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 16:15:05.394665 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":122,"threads_started":1}
I20260924 16:15:05.395907 18227 tablet_service.cc:1514] Processing DeleteTablet for tablet ea603e5f59be41a8ad5ff77be07f164c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:58012
I20260924 16:15:05.396804 18304 tablet_replica.cc:330] stopping tablet replica
I20260924 16:15:05.397107 18304 raft_consensus.cc:2229] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:05.397363 18304 raft_consensus.cc:2258] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:05.398406 18304 ts_tablet_manager.cc:1905] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:05.402102 18304 ts_tablet_manager.cc:1918] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:05.402225 18304 log.cc:1196] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal/wals/ea603e5f59be41a8ad5ff77be07f164c
I20260924 16:15:05.402534 18304 ts_tablet_manager.cc:1939] T ea603e5f59be41a8ad5ff77be07f164c P 81b6fe4c1ee2410abd349c07ef551975: Deleting consensus metadata
I20260924 16:15:05.404253 18087 catalog_manager.cc:4841] TS 81b6fe4c1ee2410abd349c07ef551975 (127.1.253.65:40807): tablet ea603e5f59be41a8ad5ff77be07f164c (table test-table [id=a1cde93e59de4e548650e34c2d0f38e3]) successfully deleted
W20260924 16:15:05.486094 18298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:05.486429 18298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:05.486516 18298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:05.490598 18298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:05.490665 18298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:05.490784 18298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:15:05.496315 18298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:46797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=39257
--tserver_master_addrs=127.1.253.126:38839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:05.497517 18298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:05.498739 18298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:15:05.509294 18298 server_base.cc:1034] running on GCE node
W20260924 16:15:05.509303 18309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:05.509274 18312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:05.509301 18310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:05.509985 18298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:05.510711 18298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:05.511938 18298 hybrid_clock.cc:648] HybridClock initialized: now 1790266505511918 us; error 30 us; skew 500 ppm
Sep 24 16:15:05 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266505, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:05.515074 18298 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:15:05.516103 18298 webserver.cc:460] Webserver started at http://127.1.253.66:39257/ using document root <none> and password file <none>
I20260924 16:15:05.516703 18298 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:05.516825 18298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:05.520886 18298 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 16:15:05.523006 18319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:05.524259 18298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:15:05.524410 18298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "a2a0c07dbed84961aa6580b1b662b221"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "0d132c0bcdf69d0bdb3a6acb4230edaf"
server_key_iv: "a1983456f3946248b41c96f6f4d8e05c"
server_key_version: "encryptionkey@0"
I20260924 16:15:05.524968 18298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:05.544667 18298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:05.548298 18298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:05.548544 18298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:05.549208 18298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:05.550139 18298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:05.550218 18298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:05.550324 18298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:05.550383 18298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:05.560257 18298 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:46797
I20260924 16:15:05.560279 18434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:46797 every 8 connection(s)
I20260924 16:15:05.561376 18298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:05.565479 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 18298
I20260924 16:15:05.565914 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:37625
--local_ip_for_outbound_sockets=127.1.253.67
--tserver_master_addrs=127.1.253.126:38839
--webserver_port=40059
--webserver_interface=127.1.253.67
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:15:05 dist-test-slave-6wvm krb5kdc[17505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266505, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:15:05.576433 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.563298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:35819 (local address 127.1.253.126:38839)
0924 16:15:05.563449 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:05.563453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:05.564297 (+ 844us) server_negotiation.cc:407] Connection header received
0924 16:15:05.565644 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:05.565647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:05.565714 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:05.565835 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:05.567209 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.567759 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.568959 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.569210 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.569338 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.572226 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:05.572229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:05.572231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:05.572255 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:05.574156 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:05.574724 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:05.574730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:05.574734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:05.574836 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:05.575167 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:05.575173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:05.575178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:05.575449 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:05.575570 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:05.576020 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 16:15:05.576180 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:15:05.577047 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.563684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:35819)
0924 16:15:05.564143 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:05.564184 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:05.565418 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:05.566034 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:05.566046 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:05.566546 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:05.567038 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.567054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.567911 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.567916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.568815 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.568827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.569386 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.569391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.569584 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.570215 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:05.570277 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:05.572042 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:05.574321 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:05.574327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:05.574342 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:05.574617 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:05.574939 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:05.574942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:05.574943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:05.575069 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:05.575558 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:05.575563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:05.575779 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 16:15:05.575930 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 16:15:05.576165 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":104,"threads_started":1}
I20260924 16:15:05.578210 18435 heartbeater.cc:347] Connected to a master server at 127.1.253.126:38839
I20260924 16:15:05.578497 18435 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:05.579041 18435 heartbeater.cc:511] Master 127.1.253.126:38839 requested a full tablet report, sending...
I20260924 16:15:05.580161 18101 ts_manager.cc:194] Registered new tserver with Master: a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797)
I20260924 16:15:05.580752 18101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:35819
W20260924 16:15:05.682185 18438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:05.682614 18438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:05.682698 18438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:05.687208 18438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:05.687288 18438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:05.687376 18438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:15:05.692759 18438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:37625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=40059
--tserver_master_addrs=127.1.253.126:38839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:05.693861 18438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:05.695014 18438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:05.705170 18446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:05.705170 18444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:05.705415 18438 server_base.cc:1034] running on GCE node
W20260924 16:15:05.705610 18443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:05.706084 18438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:05.706779 18438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:05.707991 18438 hybrid_clock.cc:648] HybridClock initialized: now 1790266505707971 us; error 30 us; skew 500 ppm
Sep 24 16:15:05 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266505, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:05.710873 18438 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:15:05.711774 18438 webserver.cc:460] Webserver started at http://127.1.253.67:40059/ using document root <none> and password file <none>
I20260924 16:15:05.712361 18438 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:05.712472 18438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:05.716594 18438 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:05.718780 18454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:05.720081 18438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:15:05.720233 18438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "2a19522336bb40afb12fa58f743d9367"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "bd377763b810f26d4d28dd9206bbb184"
server_key_iv: "be060443c8f5f5b1d9c2b5f136613dca"
server_key_version: "encryptionkey@0"
I20260924 16:15:05.720795 18438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:05.749296 18438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:05.753212 18438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:05.753424 18438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:05.754061 18438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:05.754935 18438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:05.755007 18438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:05.755074 18438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:05.755105 18438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:05.764082 18438 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:37625
I20260924 16:15:05.764110 18570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:37625 every 8 connection(s)
I20260924 16:15:05.765113 18438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:05.773105 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 18438
Sep 24 16:15:05 dist-test-slave-6wvm krb5kdc[17505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266505, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:15:05.779189 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.766972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:40723 (local address 127.1.253.126:38839)
0924 16:15:05.767114 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:05.767118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:05.767947 (+ 829us) server_negotiation.cc:407] Connection header received
0924 16:15:05.769170 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:05.769173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:05.769222 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:05.769299 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:05.770542 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.771087 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.772164 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.772371 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.772448 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.775265 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:05.775269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:05.775273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:05.775301 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:05.776950 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:05.777488 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:05.777491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:05.777492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:05.777565 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:05.777992 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:05.777998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:05.778002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:05.778226 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:05.778372 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:05.778866 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 16:15:05.778990 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:15:05.780369 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.767332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:40723)
0924 16:15:05.767795 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:05.767836 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:05.768966 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:05.769473 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:05.769480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:05.769844 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:05.770406 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.770423 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.771193 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.771196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.772064 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.772073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.772500 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.772503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.772655 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.773330 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:05.773360 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:05.775092 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:05.777071 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:05.777075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:05.777086 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:05.777395 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:05.777683 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:05.777689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:05.777693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:05.777883 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:05.778367 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:05.778372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:05.778591 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 16:15:05.778746 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 16:15:05.778976 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
I20260924 16:15:05.781719 18571 heartbeater.cc:347] Connected to a master server at 127.1.253.126:38839
I20260924 16:15:05.781966 18571 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:05.782612 18571 heartbeater.cc:511] Master 127.1.253.126:38839 requested a full tablet report, sending...
I20260924 16:15:05.782826 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.773820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.1.253.126:38839)
0924 16:15:05.774189 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:05.774193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:05.774207 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:15:05.774337 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:05.774341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:05.774391 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:05.774482 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:05.775323 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.776050 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.777301 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.777647 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.777756 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.779161 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:05.779165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:05.779167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:05.779192 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:05.780874 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:05.781355 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:05.781358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:05.781360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:05.781446 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:05.781869 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:05.781873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:05.781874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:05.782113 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:05.782316 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:05.782510 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 16:15:05.782629 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":86,"threads_started":1}
I20260924 16:15:05.783764 18101 ts_manager.cc:194] Registered new tserver with Master: 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625)
I20260924 16:15:05.784379 18101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:40723
I20260924 16:15:05.784579 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:15:05.788610 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.785178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.1.253.126:38839)
0924 16:15:05.785295 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:05.785299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:05.785499 (+ 200us) server_negotiation.cc:407] Connection header received
0924 16:15:05.785606 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:05.785609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:05.785733 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:05.785832 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:15:05.786083 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.786839 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.787703 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.787907 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.787980 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.788217 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:15:05.788336 (+ 119us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 16:15:05.788520 (+ 184us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53330: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 16:15:05.788691 18575 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53330: token expired
W20260924 16:15:05.789105 18022 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.253.126:38839)
I20260924 16:15:05.796010 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.789459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.1.253.126:38839)
0924 16:15:05.789581 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:05.789591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:05.789605 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:15:05.789699 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:05.789702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:05.789736 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:05.789800 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:05.790417 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.790875 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.791697 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.791889 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.791963 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.793273 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:05.793275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:05.793277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:05.793294 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:05.794639 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:05.795018 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:05.795021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:05.795022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:05.795093 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:05.795330 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:05.795333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:05.795334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:05.795490 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:05.795598 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:05.795802 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 16:15:05.795893 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:15:05.798601 18101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:15:05.802007 18101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:15:05.810423 18227 tablet_service.cc:1467] Processing CreateTablet for tablet 687ecfb1284f488e9ff0ac1932353b8e (DEFAULT_TABLE table=test-table [id=9df5d376bb8d4261853bb0daed9e3eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:05.811157 18227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687ecfb1284f488e9ff0ac1932353b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:05.814432 18185 tablet_bootstrap.cc:492] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975: Bootstrap starting.
I20260924 16:15:05.814551 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.808948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:37625 (local address 127.0.0.1:38594)
0924 16:15:05.809352 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:05.809368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:05.809446 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:05.810175 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:05.810177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:05.810193 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:05.810320 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.810327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.811702 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.811708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.812920 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.812932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.814038 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.814042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.814146 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.814200 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 16:15:05.814291 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:15:05.814394 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312}
I20260924 16:15:05.815250 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.809691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:46797 (local address 127.0.0.1:41596)
0924 16:15:05.809920 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:05.809935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:05.810025 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:05.811123 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:05.811126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:05.811143 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:05.811231 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.811236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.812393 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.812396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.813265 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.813276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.814839 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.814842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.814911 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.814931 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 16:15:05.815029 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 16:15:05.815109 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260924 16:15:05.815280 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.809159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38594 (local address 127.1.253.67:37625)
0924 16:15:05.809604 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:05.809624 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:05.809646 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:15:05.809716 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:05.809720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:05.809984 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:05.810207 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:05.810416 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.811559 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.813070 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.813884 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.814822 (+ 938us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.814972 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:05.815029 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 16:15:05.815132 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":158,"threads_started":1}
I20260924 16:15:05.815440 18185 tablet_bootstrap.cc:654] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:05.815577 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.809894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.1.253.66:46797)
0924 16:15:05.810436 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:05.810451 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:05.810476 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:15:05.810551 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:05.810556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:05.810753 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:05.810943 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:05.811353 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.812258 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.813404 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.814446 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.814589 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.814764 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:05.815074 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 16:15:05.815174 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":137,"threads_started":1}
I20260924 16:15:05.817121 18185 tablet_bootstrap.cc:492] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975: No bootstrap required, opened a new log
I20260924 16:15:05.817246 18185 ts_tablet_manager.cc:1397] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:15:05.817736 18185 raft_consensus.cc:348] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:05.817857 18185 raft_consensus.cc:374] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:05.817895 18185 raft_consensus.cc:729] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b6fe4c1ee2410abd349c07ef551975, State: Initialized, Role: FOLLOWER
I20260924 16:15:05.818032 18185 consensus_queue.cc:260] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:05.818344 18185 ts_tablet_manager.cc:1428] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 16:15:05.818804 18503 tablet_service.cc:1467] Processing CreateTablet for tablet 687ecfb1284f488e9ff0ac1932353b8e (DEFAULT_TABLE table=test-table [id=9df5d376bb8d4261853bb0daed9e3eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:05.819334 18368 tablet_service.cc:1467] Processing CreateTablet for tablet 687ecfb1284f488e9ff0ac1932353b8e (DEFAULT_TABLE table=test-table [id=9df5d376bb8d4261853bb0daed9e3eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:05.820048 18503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687ecfb1284f488e9ff0ac1932353b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:05.820488 18368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687ecfb1284f488e9ff0ac1932353b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:05.827427 18584 tablet_bootstrap.cc:492] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367: Bootstrap starting.
I20260924 16:15:05.827966 18585 tablet_bootstrap.cc:492] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221: Bootstrap starting.
I20260924 16:15:05.829829 18584 tablet_bootstrap.cc:654] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:05.830497 18585 tablet_bootstrap.cc:654] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:05.831025 18584 log.cc:824] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:05.831197 18585 log.cc:824] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:05.832984 18585 tablet_bootstrap.cc:492] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221: No bootstrap required, opened a new log
I20260924 16:15:05.832975 18584 tablet_bootstrap.cc:492] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367: No bootstrap required, opened a new log
I20260924 16:15:05.833213 18584 ts_tablet_manager.cc:1397] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 16:15:05.833206 18585 ts_tablet_manager.cc:1397] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:15:05.836148 18585 raft_consensus.cc:348] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:05.836146 18584 raft_consensus.cc:348] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:05.836360 18585 raft_consensus.cc:374] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:05.836391 18584 raft_consensus.cc:374] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:05.836393 18585 raft_consensus.cc:729] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a0c07dbed84961aa6580b1b662b221, State: Initialized, Role: FOLLOWER
I20260924 16:15:05.836467 18584 raft_consensus.cc:729] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a19522336bb40afb12fa58f743d9367, State: Initialized, Role: FOLLOWER
I20260924 16:15:05.836875 18585 consensus_queue.cc:260] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:05.836894 18584 consensus_queue.cc:260] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:05.837530 18435 heartbeater.cc:503] Master 127.1.253.126:38839 was elected leader, sending a full tablet report...
I20260924 16:15:05.837534 18571 heartbeater.cc:503] Master 127.1.253.126:38839 was elected leader, sending a full tablet report...
I20260924 16:15:05.837885 18585 ts_tablet_manager.cc:1428] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 16:15:05.837885 18584 ts_tablet_manager.cc:1428] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
W20260924 16:15:05.855759 18295 tablet.cc:2369] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:15:05.936592 18301 raft_consensus.cc:482] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:05.936861 18301 raft_consensus.cc:504] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:05.938158 18301 leader_election.cc:288] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625), a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797)
I20260924 16:15:05.942176 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.938548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:55301 (local address 127.1.253.67:37625)
0924 16:15:05.938695 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:05.938700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:05.939082 (+ 382us) server_negotiation.cc:407] Connection header received
0924 16:15:05.939238 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:05.939241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:05.939291 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:05.939374 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:05.939620 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.940111 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.941156 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.941557 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.941688 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.941740 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:05.941963 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 16:15:05.942042 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:15:05.942184 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.938607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:33471 (local address 127.1.253.66:46797)
0924 16:15:05.938741 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:05.938746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:05.938928 (+ 182us) server_negotiation.cc:407] Connection header received
0924 16:15:05.939138 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:05.939141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:05.939186 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:05.939275 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:05.939558 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.940046 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.941147 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.941533 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.941667 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.941723 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:05.941971 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 16:15:05.942051 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:15:05.942276 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.938534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:37625 (local address 127.1.253.65:55301)
0924 16:15:05.938982 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:05.938993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:05.939119 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:05.939388 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:05.939391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:05.939408 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:05.939507 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.939512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.940232 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.940235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.941020 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.941030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.941685 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.941689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.941810 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.941848 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:15:05.941945 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:15:05.942087 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":32,"thread_start_us":95,"threads_started":1}
I20260924 16:15:05.942301 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.938493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:46797 (local address 127.1.253.65:33471)
0924 16:15:05.938832 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:05.938843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:05.938990 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:05.939288 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:05.939290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:05.939307 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:05.939439 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.939445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.940162 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.940167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.941009 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:05.941018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.941664 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.941668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:05.941826 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.941863 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:15:05.941954 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:15:05.942092 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":115,"thread_start_us":96,"threads_started":1}
I20260924 16:15:05.943080 18523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687ecfb1284f488e9ff0ac1932353b8e" candidate_uuid: "81b6fe4c1ee2410abd349c07ef551975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a19522336bb40afb12fa58f743d9367" is_pre_election: true
I20260924 16:15:05.943086 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687ecfb1284f488e9ff0ac1932353b8e" candidate_uuid: "81b6fe4c1ee2410abd349c07ef551975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a0c07dbed84961aa6580b1b662b221" is_pre_election: true
I20260924 16:15:05.943408 18389 raft_consensus.cc:2454] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b6fe4c1ee2410abd349c07ef551975 in term 0.
I20260924 16:15:05.943401 18523 raft_consensus.cc:2454] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b6fe4c1ee2410abd349c07ef551975 in term 0.
I20260924 16:15:05.943939 18182 leader_election.cc:302] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a19522336bb40afb12fa58f743d9367, 81b6fe4c1ee2410abd349c07ef551975, a2a0c07dbed84961aa6580b1b662b221; no voters:
I20260924 16:15:05.944231 18301 raft_consensus.cc:2781] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:15:05.944305 18301 raft_consensus.cc:482] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:15:05.944335 18301 raft_consensus.cc:3037] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:05.945428 18301 raft_consensus.cc:504] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:05.945806 18301 leader_election.cc:288] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 1 election: Requested vote from peers 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625), a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797)
I20260924 16:15:05.946136 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687ecfb1284f488e9ff0ac1932353b8e" candidate_uuid: "81b6fe4c1ee2410abd349c07ef551975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a0c07dbed84961aa6580b1b662b221"
I20260924 16:15:05.946136 18523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687ecfb1284f488e9ff0ac1932353b8e" candidate_uuid: "81b6fe4c1ee2410abd349c07ef551975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a19522336bb40afb12fa58f743d9367"
I20260924 16:15:05.946286 18523 raft_consensus.cc:3037] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:05.946278 18389 raft_consensus.cc:3037] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:05.947551 18389 raft_consensus.cc:2454] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b6fe4c1ee2410abd349c07ef551975 in term 1.
I20260924 16:15:05.947551 18523 raft_consensus.cc:2454] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b6fe4c1ee2410abd349c07ef551975 in term 1.
I20260924 16:15:05.947891 18182 leader_election.cc:302] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a19522336bb40afb12fa58f743d9367, 81b6fe4c1ee2410abd349c07ef551975; no voters:
I20260924 16:15:05.948079 18301 raft_consensus.cc:2781] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:05.948303 18301 raft_consensus.cc:686] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 LEADER]: Becoming Leader. State: Replica: 81b6fe4c1ee2410abd349c07ef551975, State: Running, Role: LEADER
I20260924 16:15:05.948567 18301 consensus_queue.cc:237] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:05.951670 18100 catalog_manager.cc:5495] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81b6fe4c1ee2410abd349c07ef551975 (127.1.253.65). New cstate: current_term: 1 leader_uuid: "81b6fe4c1ee2410abd349c07ef551975" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } health_report { overall_health: HEALTHY } } }
I20260924 16:15:05.982513 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:05.978451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.1.253.65:40807)
0924 16:15:05.978827 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:05.978834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:05.978846 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:15:05.978925 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:05.978927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:05.978981 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:05.979125 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:15:05.979389 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.979907 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.980765 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:05.981059 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:05.981173 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:05.981462 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:15:05.981918 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:15:05.982049 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:05.982225 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 16:15:05.982378 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":154,"threads_started":1}
I20260924 16:15:05.986898 18523 raft_consensus.cc:1261] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367 [term 1 FOLLOWER]: Refusing update from remote peer 81b6fe4c1ee2410abd349c07ef551975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:05.986899 18389 raft_consensus.cc:1261] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221 [term 1 FOLLOWER]: Refusing update from remote peer 81b6fe4c1ee2410abd349c07ef551975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:05.987651 18592 consensus_queue.cc:1035] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:05.987915 18301 consensus_queue.cc:1035] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:05.997845 18596 mvcc.cc:205] Tried to move back new op lower bound from 7332931608515141632 to 7332931608368054272. Current Snapshot: MvccSnapshot[applied={T|T < 7332931608515141632}]
I20260924 16:15:06.004617 18099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:06.004803 18099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:06.007094 18099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Sending DeleteTablet for 3 replicas of tablet 687ecfb1284f488e9ff0ac1932353b8e
I20260924 16:15:06.007844 18368 tablet_service.cc:1514] Processing DeleteTablet for tablet 687ecfb1284f488e9ff0ac1932353b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:05 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:41596
I20260924 16:15:06.007843 18503 tablet_service.cc:1514] Processing DeleteTablet for tablet 687ecfb1284f488e9ff0ac1932353b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:05 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:38594
I20260924 16:15:06.008265 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 18141
I20260924 16:15:06.008287 18227 tablet_service.cc:1514] Processing DeleteTablet for tablet 687ecfb1284f488e9ff0ac1932353b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:05 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:58012
I20260924 16:15:06.009999 18605 tablet_replica.cc:330] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367: stopping tablet replica
I20260924 16:15:06.010353 18605 raft_consensus.cc:2229] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:06.010675 18605 raft_consensus.cc:2258] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:06.010964 18606 tablet_replica.cc:330] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221: stopping tablet replica
I20260924 16:15:06.011721 18606 raft_consensus.cc:2229] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:06.012040 18606 raft_consensus.cc:2258] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:06.012310 18605 ts_tablet_manager.cc:1905] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:06.013509 18606 ts_tablet_manager.cc:1905] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:06.014915 18605 ts_tablet_manager.cc:1918] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:06.015039 18605 log.cc:1196] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/687ecfb1284f488e9ff0ac1932353b8e
I20260924 16:15:06.015374 18605 ts_tablet_manager.cc:1939] T 687ecfb1284f488e9ff0ac1932353b8e P 2a19522336bb40afb12fa58f743d9367: Deleting consensus metadata
I20260924 16:15:06.016903 18087 catalog_manager.cc:4841] TS 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625): tablet 687ecfb1284f488e9ff0ac1932353b8e (table test-table [id=9df5d376bb8d4261853bb0daed9e3eae]) successfully deleted
I20260924 16:15:06.017050 18606 ts_tablet_manager.cc:1918] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:06.017155 18606 log.cc:1196] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/687ecfb1284f488e9ff0ac1932353b8e
I20260924 16:15:06.017510 18606 ts_tablet_manager.cc:1939] T 687ecfb1284f488e9ff0ac1932353b8e P a2a0c07dbed84961aa6580b1b662b221: Deleting consensus metadata
W20260924 16:15:06.017980 18087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:40807 (error 108)
I20260924 16:15:06.018378 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 18298
I20260924 16:15:06.019955 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:06.019559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40807 (local address 127.0.0.1:58042)
0924 16:15:06.019767 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:06.019846 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260924 16:15:06.020149 18087 catalog_manager.cc:4584] TS 81b6fe4c1ee2410abd349c07ef551975 (127.1.253.65:40807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 687ecfb1284f488e9ff0ac1932353b8e: Network error: Client connection negotiation failed: client connection to 127.1.253.65:40807: connect: Connection refused (error 111)
I20260924 16:15:06.025874 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 18438
I20260924 16:15:06.027207 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:06.026003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:46797 (local address 127.0.0.1:41598)
0924 16:15:06.027033 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:06.027111 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.66:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":925}
I20260924 16:15:06.031852 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 18065
I20260924 16:15:26.039296 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:38839
--webserver_interface=127.1.253.126
--webserver_port=41301
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:38839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:26.158351 18608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:26.158651 18608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:26.158730 18608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:26.162824 18608 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:15:26.162912 18608 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:15:26.162940 18608 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:15:26.162971 18608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:26.163038 18608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:26.163082 18608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:26.163120 18608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:26.163151 18608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:26.168555 18608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:38839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:38839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=41301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:26.169754 18608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:26.170873 18608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:26.179925 18613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:26.179910 18614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:26.179915 18616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:26.180037 18608 server_base.cc:1034] running on GCE node
I20260924 16:15:26.180739 18608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:26.181850 18608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:26.183044 18608 hybrid_clock.cc:648] HybridClock initialized: now 1790266526183027 us; error 31 us; skew 500 ppm
Sep 24 16:15:26 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266526, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:26.193565 18608 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:15:26.194639 18608 webserver.cc:460] Webserver started at http://127.1.253.126:41301/ using document root <none> and password file <none>
I20260924 16:15:26.195253 18608 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:26.195345 18608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:26.199359 18608 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:26.201700 18623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:26.203137 18608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:26.203297 18608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "aff9e1fcd06d4b0a97d783de156a6b2c"
format_stamp: "Formatted at 2026-09-24 16:14:43 on dist-test-slave-6wvm"
server_key: "e81edce74934470dc77c018def8fa0bd"
server_key_iv: "22db76418ae0f0ebed84adc8ba619057"
server_key_version: "encryptionkey@0"
I20260924 16:15:26.203879 18608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:26.242012 18608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:26.245245 18608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:26.245457 18608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:26.252697 18608 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:38839
I20260924 16:15:26.253031 18676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:38839 every 8 connection(s)
I20260924 16:15:26.253823 18608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:26.257007 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 18608
I20260924 16:15:26.257503 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:40807
--local_ip_for_outbound_sockets=127.1.253.65
--tserver_master_addrs=127.1.253.126:38839
--webserver_port=36101
--webserver_interface=127.1.253.65
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:15:26.258441 18677 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:15:26.260113 18677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Bootstrap starting.
I20260924 16:15:26.278183 18677 log.cc:824] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:26.289889 18677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:15:26.290412 18677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Bootstrap complete.
I20260924 16:15:26.294647 18677 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } }
I20260924 16:15:26.295097 18677 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff9e1fcd06d4b0a97d783de156a6b2c, State: Initialized, Role: FOLLOWER
I20260924 16:15:26.295670 18677 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } }
I20260924 16:15:26.295840 18677 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:26.295928 18677 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:26.296051 18677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:15:26.298012 18677 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } }
I20260924 16:15:26.298420 18677 leader_election.cc:302] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff9e1fcd06d4b0a97d783de156a6b2c; no voters:
I20260924 16:15:26.298760 18677 leader_election.cc:288] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 16:15:26.298928 18682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 16:15:26.299188 18682 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [term 3 LEADER]: Becoming Leader. State: Replica: aff9e1fcd06d4b0a97d783de156a6b2c, State: Running, Role: LEADER
I20260924 16:15:26.299561 18682 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } }
I20260924 16:15:26.300169 18677 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:26.301091 18683 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } } }
I20260924 16:15:26.301227 18683 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:26.302095 18691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:26.302377 18684 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff9e1fcd06d4b0a97d783de156a6b2c. Latest consensus state: current_term: 3 leader_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9e1fcd06d4b0a97d783de156a6b2c" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 38839 } } }
I20260924 16:15:26.302788 18684 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:26.307848 18691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:26.308490 18691 catalog_manager.cc:1259] Loaded cluster ID: dcbffd456b6a43b5b94669609c9a5432
I20260924 16:15:26.308549 18691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:26.311035 18691 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:26.311758 18691 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Loaded TSK: 0
I20260924 16:15:26.311841 18691 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aff9e1fcd06d4b0a97d783de156a6b2c: Loaded TSK: 1
F20260924 16:15:26.312563 18691 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f43457529ad google::LogMessage::Fail() at ??:0
@ 0x7f434575588a google::LogMessage::SendToLog() at ??:0
@ 0x7f43457524de google::LogMessage::Flush() at ??:0
@ 0x7f4345753a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f434841618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f434840ef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f43500bb7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f435009599c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4350096178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f43500d7df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f43500f682b std::function<>::operator()() at ??:0
@ 0x7f4350095b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4350096f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4350093579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f43500d7640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f43479a43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f43479a4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f43479a63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4347995512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4348fd96db start_thread at ??:0
@ 0x7f434425061f clone at ??:0
W20260924 16:15:26.416083 18680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:26.416342 18680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:26.416385 18680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:26.421087 18680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:26.421173 18680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:26.421263 18680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:15:26.427354 18680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:40807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=36101
--tserver_master_addrs=127.1.253.126:38839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:26.428516 18680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:26.429615 18680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:26.440112 18728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:26.440131 18726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:26.440388 18725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:26.440258 18680 server_base.cc:1034] running on GCE node
I20260924 16:15:26.440881 18680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:26.441586 18680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:26.442770 18680 hybrid_clock.cc:648] HybridClock initialized: now 1790266526442731 us; error 54 us; skew 500 ppm
Sep 24 16:15:26 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266526, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:26.445695 18680 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:15:26.446748 18680 webserver.cc:460] Webserver started at http://127.1.253.65:36101/ using document root <none> and password file <none>
I20260924 16:15:26.447320 18680 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:26.447391 18680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:26.451442 18680 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:26.453846 18735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:26.455377 18680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:26.455546 18680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "81b6fe4c1ee2410abd349c07ef551975"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "ad39dae3edc933136396b6b112a8f8d3"
server_key_iv: "1c7ea88f3b9cc59ef802354e519d4a16"
server_key_version: "encryptionkey@0"
I20260924 16:15:26.456141 18680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:26.476321 18680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:26.479869 18680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:26.480079 18680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:26.480726 18680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:26.481892 18742 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 16:15:26.487648 18680 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:15:26.487743 18680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 16:15:26.487843 18680 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:15:26.489594 18680 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:15:26.489666 18680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 16:15:26.489998 18742 tablet_bootstrap.cc:492] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975: Bootstrap starting.
I20260924 16:15:26.500727 18680 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:40807
I20260924 16:15:26.501721 18680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:26.502797 18851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:40807 every 8 connection(s)
I20260924 16:15:26.505475 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 18680
I20260924 16:15:26.506009 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:46797
--local_ip_for_outbound_sockets=127.1.253.66
--tserver_master_addrs=127.1.253.126:38839
--webserver_port=39257
--webserver_interface=127.1.253.66
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:15:26.514308 18742 log.cc:824] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:26.522075 18742 tablet_bootstrap.cc:492] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:15:26.522712 18742 tablet_bootstrap.cc:492] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975: Bootstrap complete.
I20260924 16:15:26.523860 18742 ts_tablet_manager.cc:1397] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.010s
I20260924 16:15:26.526530 18742 raft_consensus.cc:348] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:26.527011 18742 raft_consensus.cc:729] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b6fe4c1ee2410abd349c07ef551975, State: Initialized, Role: FOLLOWER
I20260924 16:15:26.527554 18742 consensus_queue.cc:260] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:26.528638 18742 ts_tablet_manager.cc:1428] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260924 16:15:26.564321 18854 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:15:26.504003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:40013)
0924 16:15:26.504519 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:26.504576 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:26.505725 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:26.563542 (+ 57817us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":146,"threads_started":1}
W20260924 16:15:26.564985 18852 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:38839 (0 consecutive failures): Network error: Failed to ping master at 127.1.253.126:38839: Client connection negotiation failed: client connection to 127.1.253.126:38839: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 16:15:26.565757 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:26.565485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:58771)
0924 16:15:26.565620 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:26.565692 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 16:15:26.566593 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:26.566384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:40951)
0924 16:15:26.566495 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:26.566539 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:15:26.566864 18852 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:15:26.641268 18855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:26.641636 18855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:26.641710 18855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:26.645747 18855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:26.645810 18855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:26.645865 18855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:15:26.651129 18855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:46797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=39257
--tserver_master_addrs=127.1.253.126:38839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:26.652204 18855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:26.653278 18855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:26.663192 18863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:26.663208 18862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:26.663178 18865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:26.663645 18855 server_base.cc:1034] running on GCE node
I20260924 16:15:26.664076 18855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:26.664790 18855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:26.666004 18855 hybrid_clock.cc:648] HybridClock initialized: now 1790266526665988 us; error 27 us; skew 500 ppm
Sep 24 16:15:26 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266526, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:26.668802 18855 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:15:26.669740 18855 webserver.cc:460] Webserver started at http://127.1.253.66:39257/ using document root <none> and password file <none>
I20260924 16:15:26.670348 18855 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:26.670416 18855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:26.674355 18855 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:26.676548 18872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:26.677778 18855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:15:26.677937 18855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "a2a0c07dbed84961aa6580b1b662b221"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "0d132c0bcdf69d0bdb3a6acb4230edaf"
server_key_iv: "a1983456f3946248b41c96f6f4d8e05c"
server_key_version: "encryptionkey@0"
I20260924 16:15:26.678539 18855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:26.705618 18855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:26.708803 18855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:26.709043 18855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:26.709707 18855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:26.710623 18855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:26.710700 18855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:26.710778 18855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:26.710824 18855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:26.720152 18855 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:46797
I20260924 16:15:26.720175 18987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:46797 every 8 connection(s)
I20260924 16:15:26.721221 18855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:26.725327 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:26.723605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:50747)
0924 16:15:26.724113 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:26.724560 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":125,"threads_started":1}
W20260924 16:15:26.725991 18988 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:38839 (0 consecutive failures): Network error: Failed to ping master at 127.1.253.126:38839: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
I20260924 16:15:26.726027 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 18855
I20260924 16:15:26.726513 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:37625
--local_ip_for_outbound_sockets=127.1.253.67
--tserver_master_addrs=127.1.253.126:38839
--webserver_port=40059
--webserver_interface=127.1.253.67
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:15:26.726828 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:26.726522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:33485)
0924 16:15:26.726678 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:26.726746 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 16:15:26.727558 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:26.727293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:51789)
0924 16:15:26.727423 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:26.727482 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:15:26.727744 18988 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:15:26.840245 18991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:26.840510 18991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:26.840554 18991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:26.844614 18991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:26.844673 18991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:26.844784 18991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:15:26.850070 18991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:37625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=40059
--tserver_master_addrs=127.1.253.126:38839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:26.851209 18991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:26.852272 18991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:15:26.862493 18991 server_base.cc:1034] running on GCE node
W20260924 16:15:26.862438 18999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:26.862468 18996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:26.862438 18997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:26.863112 18991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:26.863852 18991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:26.865096 18991 hybrid_clock.cc:648] HybridClock initialized: now 1790266526865063 us; error 50 us; skew 500 ppm
Sep 24 16:15:26 dist-test-slave-6wvm krb5kdc[17505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266526, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:26.868249 18991 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:15:26.869172 18991 webserver.cc:460] Webserver started at http://127.1.253.67:40059/ using document root <none> and password file <none>
I20260924 16:15:26.869781 18991 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:26.869844 18991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:26.874092 18991 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:26.876544 19006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:26.877930 18991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:15:26.878085 18991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "2a19522336bb40afb12fa58f743d9367"
format_stamp: "Formatted at 2026-09-24 16:14:44 on dist-test-slave-6wvm"
server_key: "bd377763b810f26d4d28dd9206bbb184"
server_key_iv: "be060443c8f5f5b1d9c2b5f136613dca"
server_key_version: "encryptionkey@0"
I20260924 16:15:26.878757 18991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:26.896440 18991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:26.900074 18991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:26.900285 18991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:26.900959 18991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:26.901836 18991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:26.901919 18991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:26.901994 18991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:26.902038 18991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:26.911159 18991 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:37625
I20260924 16:15:26.911186 19121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:37625 every 8 connection(s)
I20260924 16:15:26.912158 18991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:26.914075 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 18991
I20260924 16:15:26.916359 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:26.914556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:55609)
0924 16:15:26.915048 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:26.915642 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":122,"threads_started":1}
W20260924 16:15:26.916836 19122 heartbeater.cc:644] Failed to heartbeat to 127.1.253.126:38839 (0 consecutive failures): Network error: Failed to ping master at 127.1.253.126:38839: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
I20260924 16:15:26.917495 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:26.917229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:43047)
0924 16:15:26.917378 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:26.917440 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 16:15:26.918102 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:26.917893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:45955)
0924 16:15:26.918002 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:26.918054 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 16:15:26.918330 19122 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:15:27.567932 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:27.567485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:45945)
0924 16:15:27.567747 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:27.567803 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260924 16:15:27.728902 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:27.728410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:39903)
0924 16:15:27.728683 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:27.728742 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260924 16:15:27.919257 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:27.918851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:32847)
0924 16:15:27.919102 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:27.919158 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20260924 16:15:27.930660 19129 raft_consensus.cc:482] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:27.930922 19129 raft_consensus.cc:504] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:27.932117 19129 leader_election.cc:288] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625), a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797)
Sep 24 16:15:27 dist-test-slave-6wvm krb5kdc[17505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266526, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.66@KRBTEST.COM
Sep 24 16:15:27 dist-test-slave-6wvm krb5kdc[17505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266526, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.67@KRBTEST.COM
I20260924 16:15:27.947816 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:27.932631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:41067 (local address 127.1.253.66:46797)
0924 16:15:27.932934 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:27.932957 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:27.932992 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:15:27.933922 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:27.933932 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:27.934444 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:27.934800 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:27.935907 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:27.936892 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:27.938279 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:27.939000 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:27.939172 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:27.942046 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:27.942052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:27.942066 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:27.942097 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:27.944291 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:27.945127 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:27.945133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:27.945140 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:27.945311 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:27.945768 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:27.945772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:27.945774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:27.946356 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:27.946492 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:27.947308 (+ 816us) server_negotiation.cc:299] Negotiation successful
0924 16:15:27.947624 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":59,"threads_started":1}
I20260924 16:15:27.947813 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:27.932462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:35081 (local address 127.1.253.67:37625)
0924 16:15:27.932892 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:27.932922 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:27.932973 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:15:27.933922 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:27.933932 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:27.934444 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:27.934800 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:27.935919 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:27.936901 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:27.938284 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:27.939000 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:27.939172 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:27.942361 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:27.942370 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:27.942387 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:27.942428 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:27.944291 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:27.945127 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:27.945133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:27.945140 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:27.945311 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:27.945768 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:27.945772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:27.945774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:27.946356 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:27.946492 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:27.947308 (+ 816us) server_negotiation.cc:299] Negotiation successful
0924 16:15:27.947624 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":69,"threads_started":1}
I20260924 16:15:27.947790 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:27.932395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:46797 (local address 127.1.253.65:41067)
0924 16:15:27.932778 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:27.932792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:27.932910 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:27.935315 (+ 2405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:27.935317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:27.935492 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:27.935714 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:27.935729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:27.937071 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:27.937082 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:27.938093 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:27.938105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:27.939379 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:27.939382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:27.939447 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:27.940131 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:27.940143 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:27.941779 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:27.944449 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:27.944456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:27.944474 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:27.944992 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:27.945434 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:27.945437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:27.945439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:27.945663 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:27.946511 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:27.946519 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:27.946911 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 16:15:27.947157 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 16:15:27.947567 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":229,"spinlock_wait_cycles":16128,"thread_start_us":99,"threads_started":1}
I20260924 16:15:27.947789 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:27.932312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:37625 (local address 127.1.253.65:35081)
0924 16:15:27.932542 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:27.932561 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:27.932675 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:27.934865 (+ 2190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:27.934870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:27.935248 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:27.935714 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:27.935728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:27.937070 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:27.937073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:27.938095 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:27.938106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:27.939133 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:27.939136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:27.939279 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:27.939987 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:27.940016 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:27.942129 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:27.944449 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:27.944456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:27.944474 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:27.944992 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:27.945433 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:27.945437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:27.945439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:27.945663 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:27.946511 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:27.946519 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:27.946911 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 16:15:27.947157 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 16:15:27.947620 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":204,"spinlock_wait_cycles":33152}
W20260924 16:15:27.951085 18736 leader_election.cc:341] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797): Not found: Tablet not found: 687ecfb1284f488e9ff0ac1932353b8e
W20260924 16:15:27.951359 18739 leader_election.cc:341] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625): Not found: Tablet not found: 687ecfb1284f488e9ff0ac1932353b8e
I20260924 16:15:27.951434 18739 leader_election.cc:302] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81b6fe4c1ee2410abd349c07ef551975; no voters: 2a19522336bb40afb12fa58f743d9367, a2a0c07dbed84961aa6580b1b662b221
I20260924 16:15:27.951771 19129 raft_consensus.cc:2726] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:15:28.569357 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:28.568751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:40235)
0924 16:15:28.569121 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:28.569213 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":89,"threads_started":1}
I20260924 16:15:28.730347 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:28.729819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:49375)
0924 16:15:28.730109 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:28.730170 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":89,"threads_started":1}
I20260924 16:15:28.920523 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:28.920104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:46275)
0924 16:15:28.920367 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:28.920430 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260924 16:15:29.570804 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.570331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:57255)
0924 16:15:29.570592 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.570677 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
I20260924 16:15:29.731715 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.731234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:34745)
0924 16:15:29.731534 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.731613 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260924 16:15:29.881244 19138 raft_consensus.cc:482] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:29.881443 19138 raft_consensus.cc:504] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:29.881875 19138 leader_election.cc:288] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625), a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797)
W20260924 16:15:29.882766 18736 leader_election.cc:341] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797): Not found: Tablet not found: 687ecfb1284f488e9ff0ac1932353b8e
W20260924 16:15:29.882869 18739 leader_election.cc:341] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625): Not found: Tablet not found: 687ecfb1284f488e9ff0ac1932353b8e
I20260924 16:15:29.882969 18739 leader_election.cc:302] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81b6fe4c1ee2410abd349c07ef551975; no voters: 2a19522336bb40afb12fa58f743d9367, a2a0c07dbed84961aa6580b1b662b221
I20260924 16:15:29.883106 19138 raft_consensus.cc:2726] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:15:29.921748 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.921314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:58385)
0924 16:15:29.921599 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.921658 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
I20260924 16:15:30.572050 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:30.571586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:54833)
0924 16:15:30.571880 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:30.571951 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260924 16:15:30.732892 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:30.732477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:60653)
0924 16:15:30.732737 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:30.732794 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260924 16:15:30.923167 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:30.922699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:46293)
0924 16:15:30.922992 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:30.923056 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
I20260924 16:15:31.555121 19143 raft_consensus.cc:482] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:31.555266 19143 raft_consensus.cc:504] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:31.555784 19143 leader_election.cc:288] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625), a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797)
W20260924 16:15:31.556697 18736 leader_election.cc:341] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797): Not found: Tablet not found: 687ecfb1284f488e9ff0ac1932353b8e
W20260924 16:15:31.556798 18739 leader_election.cc:341] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625): Not found: Tablet not found: 687ecfb1284f488e9ff0ac1932353b8e
I20260924 16:15:31.556841 18736 leader_election.cc:302] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81b6fe4c1ee2410abd349c07ef551975; no voters: 2a19522336bb40afb12fa58f743d9367, a2a0c07dbed84961aa6580b1b662b221
I20260924 16:15:31.556980 19143 raft_consensus.cc:2726] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:15:31.573609 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:31.573141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:55487)
0924 16:15:31.573451 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:31.573513 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":95,"threads_started":1}
I20260924 16:15:31.734351 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:31.733708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:50261)
0924 16:15:31.734022 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:31.734091 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":92,"threads_started":1}
I20260924 16:15:31.924381 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:31.923914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:35935)
0924 16:15:31.924224 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:31.924284 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":91,"threads_started":1}
I20260924 16:15:32.575088 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:32.574612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:41127)
0924 16:15:32.574918 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:32.574980 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260924 16:15:32.735672 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:32.735192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:50683)
0924 16:15:32.735508 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:32.735572 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":104,"threads_started":1}
I20260924 16:15:32.925525 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:32.925115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:58329)
0924 16:15:32.925375 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:32.925436 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260924 16:15:33.576556 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:33.576050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:42685)
0924 16:15:33.576349 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:33.576407 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":101,"threads_started":1}
I20260924 16:15:33.737030 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:33.736525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:46101)
0924 16:15:33.736853 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:33.736922 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":76,"threads_started":1}
I20260924 16:15:33.927007 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:33.926489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:33601)
0924 16:15:33.926838 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:33.926901 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":96,"threads_started":1}
I20260924 16:15:34.577996 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:34.577541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:43073)
0924 16:15:34.577831 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:34.577892 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
I20260924 16:15:34.738370 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:34.737877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:57385)
0924 16:15:34.738149 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:34.738208 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260924 16:15:34.928318 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:34.927805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:37585)
0924 16:15:34.928156 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:34.928220 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":107,"threads_started":1}
I20260924 16:15:35.579169 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:35.578763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:56935)
0924 16:15:35.579005 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:35.579059 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260924 16:15:35.739562 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:35.739108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:44109)
0924 16:15:35.739393 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:35.739464 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
I20260924 16:15:35.929559 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:35.929141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:49915)
0924 16:15:35.929409 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:35.929465 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
I20260924 16:15:35.953531 19159 raft_consensus.cc:482] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:35.953678 19159 raft_consensus.cc:504] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:35.954123 19159 leader_election.cc:288] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625), a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797)
W20260924 16:15:35.954944 18736 leader_election.cc:341] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797): Not found: Tablet not found: 687ecfb1284f488e9ff0ac1932353b8e
W20260924 16:15:35.955057 18739 leader_election.cc:341] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625): Not found: Tablet not found: 687ecfb1284f488e9ff0ac1932353b8e
I20260924 16:15:35.955175 18739 leader_election.cc:302] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81b6fe4c1ee2410abd349c07ef551975; no voters: 2a19522336bb40afb12fa58f743d9367, a2a0c07dbed84961aa6580b1b662b221
I20260924 16:15:35.955374 19159 raft_consensus.cc:2726] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:15:36.580546 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:36.580119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:46593)
0924 16:15:36.580375 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:36.580447 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260924 16:15:36.740875 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:36.740448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:47413)
0924 16:15:36.740715 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:36.740771 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":83,"threads_started":1}
I20260924 16:15:36.930974 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:36.930547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:47227)
0924 16:15:36.930818 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:36.930876 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
I20260924 16:15:37.582144 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.581577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:55575)
0924 16:15:37.581961 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:37.582030 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":117,"threads_started":1}
I20260924 16:15:37.742148 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.741711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:48135)
0924 16:15:37.741983 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:37.742046 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260924 16:15:37.932267 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.931800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:46953)
0924 16:15:37.932110 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:37.932169 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":119,"threads_started":1}
I20260924 16:15:38.583510 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:38.583004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:54969)
0924 16:15:38.583330 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:38.583410 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":163,"threads_started":1}
I20260924 16:15:38.743466 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:38.743015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:33579)
0924 16:15:38.743293 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:38.743362 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260924 16:15:38.933514 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:38.933120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:36055)
0924 16:15:38.933365 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:38.933420 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260924 16:15:39.212781 19169 raft_consensus.cc:482] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:39.212936 19169 raft_consensus.cc:504] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a19522336bb40afb12fa58f743d9367" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 37625 } } peers { permanent_uuid: "a2a0c07dbed84961aa6580b1b662b221" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 46797 } } peers { permanent_uuid: "81b6fe4c1ee2410abd349c07ef551975" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40807 } }
I20260924 16:15:39.213459 19169 leader_election.cc:288] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625), a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797)
W20260924 16:15:39.214370 18739 leader_election.cc:341] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a19522336bb40afb12fa58f743d9367 (127.1.253.67:37625): Not found: Tablet not found: 687ecfb1284f488e9ff0ac1932353b8e
W20260924 16:15:39.214481 18736 leader_election.cc:341] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2a0c07dbed84961aa6580b1b662b221 (127.1.253.66:46797): Not found: Tablet not found: 687ecfb1284f488e9ff0ac1932353b8e
I20260924 16:15:39.214548 18736 leader_election.cc:302] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81b6fe4c1ee2410abd349c07ef551975; no voters: 2a19522336bb40afb12fa58f743d9367, a2a0c07dbed84961aa6580b1b662b221
I20260924 16:15:39.214710 19169 raft_consensus.cc:2726] T 687ecfb1284f488e9ff0ac1932353b8e P 81b6fe4c1ee2410abd349c07ef551975 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:15:39.584640 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:39.584245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:46817)
0924 16:15:39.584488 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:39.584541 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260924 16:15:39.744843 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:39.744421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:51371)
0924 16:15:39.744684 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:39.744745 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260924 16:15:39.934979 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:39.934503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:55197)
0924 16:15:39.934824 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:39.934883 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":142,"threads_started":1}
I20260924 16:15:40.586134 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.585491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:42717)
0924 16:15:40.585960 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.586033 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":205,"threads_started":1}
I20260924 16:15:40.746327 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.745811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:49401)
0924 16:15:40.746107 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.746167 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
I20260924 16:15:40.936533 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.936011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.67:55569)
0924 16:15:40.936308 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.936412 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":98,"threads_started":1}
I20260924 16:15:41.587590 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:41.587175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.65:56951)
0924 16:15:41.587438 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:41.587493 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260924 16:15:41.747725 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:41.747255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:38839 (local address 127.1.253.66:53913)
0924 16:15:41.747561 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:41.747621 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.126:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":86,"threads_started":1}
I20260924 16:15:41.922781 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 18680
I20260924 16:15:41.930590 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 18855
I20260924 16:15:41.936968 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 18991
2026-09-24T16:15:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59460 ms)
[----------] 30 tests from SecurityITest (244684 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:42 dist-test-slave-6wvm krb5kdc[19180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:42 dist-test-slave-6wvm krb5kdc[19180](info): set up 2 sockets
Sep 24 16:15:42 dist-test-slave-6wvm krb5kdc[19180](info): commencing operation
krb5kdc: starting...
W20260924 16:15:42.997684 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:15:43 dist-test-slave-6wvm krb5kdc[19180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:15:43.126003 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:41305
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:41305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:43.244261 19193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:43.244541 19193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:43.244581 19193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:43.248574 19193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:43.248625 19193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:43.248644 19193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:43.248679 19193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:43.248693 19193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:43.254068 19193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:41305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:41305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:43.255169 19193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:43.256280 19193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:43.265111 19198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:43.265107 19201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:43.265107 19199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:43.265177 19193 server_base.cc:1034] running on GCE node
I20260924 16:15:43.265959 19193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:43.267148 19193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:43.268348 19193 hybrid_clock.cc:648] HybridClock initialized: now 1790266543268328 us; error 31 us; skew 500 ppm
Sep 24 16:15:43 dist-test-slave-6wvm krb5kdc[19180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266543, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:43.271178 19193 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:15:43.272145 19193 webserver.cc:460] Webserver started at http://127.1.253.126:36141/ using document root <none> and password file <none>
I20260924 16:15:43.272760 19193 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:43.272830 19193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:43.273080 19193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:43.275256 19193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "6cb8b9239ca748f485dafe5fe3533a22"
format_stamp: "Formatted at 2026-09-24 16:15:43 on dist-test-slave-6wvm"
server_key: "e4e33e35bca235fdccf79da069ee4889"
server_key_iv: "8cd0f7e366cf60aa7aaea090c76ac6e1"
server_key_version: "encryptionkey@0"
I20260924 16:15:43.275842 19193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "6cb8b9239ca748f485dafe5fe3533a22"
format_stamp: "Formatted at 2026-09-24 16:15:43 on dist-test-slave-6wvm"
server_key: "e4e33e35bca235fdccf79da069ee4889"
server_key_iv: "8cd0f7e366cf60aa7aaea090c76ac6e1"
server_key_version: "encryptionkey@0"
I20260924 16:15:43.279608 19193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:43.282017 19208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:43.283428 19193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:15:43.283531 19193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "6cb8b9239ca748f485dafe5fe3533a22"
format_stamp: "Formatted at 2026-09-24 16:15:43 on dist-test-slave-6wvm"
server_key: "e4e33e35bca235fdccf79da069ee4889"
server_key_iv: "8cd0f7e366cf60aa7aaea090c76ac6e1"
server_key_version: "encryptionkey@0"
I20260924 16:15:43.283619 19193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:43.301949 19193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:43.305150 19193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:43.305310 19193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:43.312357 19193 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:41305
I20260924 16:15:43.312454 19261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:41305 every 8 connection(s)
I20260924 16:15:43.313410 19193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:43.316537 19262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:43.322396 19262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22: Bootstrap starting.
I20260924 16:15:43.323241 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 19193
I20260924 16:15:43.323391 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:15:43.323724 2037 external_mini_cluster.cc:1428] Setting key cec9141f96881fd7e6ddb78a43c462a3
I20260924 16:15:43.325693 19262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:43.326558 19262 log.cc:824] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:43.328873 19262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22: No bootstrap required, opened a new log
Sep 24 16:15:43 dist-test-slave-6wvm krb5kdc[19180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:15:43.333628 19262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb8b9239ca748f485dafe5fe3533a22" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41305 } }
I20260924 16:15:43.333878 19262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:43.333945 19262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb8b9239ca748f485dafe5fe3533a22, State: Initialized, Role: FOLLOWER
I20260924 16:15:43.334544 19262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb8b9239ca748f485dafe5fe3533a22" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41305 } }
I20260924 16:15:43.334743 19262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:43.334820 19262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:43.334924 19262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:43.335922 19262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb8b9239ca748f485dafe5fe3533a22" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41305 } }
I20260924 16:15:43.336285 19262 leader_election.cc:302] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cb8b9239ca748f485dafe5fe3533a22; no voters:
I20260924 16:15:43.336681 19262 leader_election.cc:288] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:43.336761 19267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:43.337004 19267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [term 1 LEADER]: Becoming Leader. State: Replica: 6cb8b9239ca748f485dafe5fe3533a22, State: Running, Role: LEADER
I20260924 16:15:43.337358 19267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb8b9239ca748f485dafe5fe3533a22" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41305 } }
I20260924 16:15:43.338109 19262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:43.339277 19268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cb8b9239ca748f485dafe5fe3533a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb8b9239ca748f485dafe5fe3533a22" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41305 } } }
I20260924 16:15:43.339407 19268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:43.339489 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.325158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.1.253.126:41305)
0924 16:15:43.325607 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:43.325634 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:43.325672 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:15:43.326461 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:43.326480 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:43.326904 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:43.327329 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:43.328189 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.329246 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.330423 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.330880 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.330997 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:43.333973 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:43.333981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:43.334001 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:43.334043 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:43.335988 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:43.336479 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:43.336485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:43.336491 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:43.336625 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:43.336913 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:43.336916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:43.336918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:43.337471 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:43.337745 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:43.338035 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 16:15:43.338432 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":160,"threads_started":1}
I20260924 16:15:43.340520 19269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb8b9239ca748f485dafe5fe3533a22. Latest consensus state: current_term: 1 leader_uuid: "6cb8b9239ca748f485dafe5fe3533a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb8b9239ca748f485dafe5fe3533a22" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41305 } } }
I20260924 16:15:43.340725 19269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22 [sys.catalog]: This master's current role is: LEADER
W20260924 16:15:43.343883 19283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:15:43.343961 19283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:15:43.344020 19276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:43.344724 19276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:43.350553 19276 catalog_manager.cc:1347] Generated new cluster ID: 8da8a86a0c584347b1c25f06241a5d37
I20260924 16:15:43.350626 19276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:43.378665 19276 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:43.379983 19276 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:43.394690 19276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22: Generated new TSK 0
I20260924 16:15:43.395468 19276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:15:43.433986 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41305
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:43.549579 19288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:43.550021 19288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:43.550117 19288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:43.554328 19288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:43.554390 19288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:43.554450 19288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:15:43.559942 19288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:43.561053 19288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:43.562125 19288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:43.572731 19297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:43.572705 19294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:43.572705 19293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:43.573076 19288 server_base.cc:1034] running on GCE node
I20260924 16:15:43.573432 19288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:43.574055 19288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:43.575227 19288 hybrid_clock.cc:648] HybridClock initialized: now 1790266543575195 us; error 45 us; skew 500 ppm
Sep 24 16:15:43 dist-test-slave-6wvm krb5kdc[19180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266543, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:43.577953 19288 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:15:43.578905 19288 webserver.cc:460] Webserver started at http://127.1.253.65:43315/ using document root <none> and password file <none>
I20260924 16:15:43.579485 19288 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:43.579583 19288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:43.579815 19288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:43.581909 19288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "d97b5470722444c1a0034a6b92f4cf6b"
format_stamp: "Formatted at 2026-09-24 16:15:43 on dist-test-slave-6wvm"
server_key: "a0ec5aabeb9f1926b361ed0fbe0e75d6"
server_key_iv: "8eb53001220a095b2127ba30baaff06c"
server_key_version: "encryptionkey@0"
I20260924 16:15:43.582515 19288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "d97b5470722444c1a0034a6b92f4cf6b"
format_stamp: "Formatted at 2026-09-24 16:15:43 on dist-test-slave-6wvm"
server_key: "a0ec5aabeb9f1926b361ed0fbe0e75d6"
server_key_iv: "8eb53001220a095b2127ba30baaff06c"
server_key_version: "encryptionkey@0"
I20260924 16:15:43.586216 19288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:15:43.589026 19304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:43.590444 19288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 16:15:43.590577 19288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "d97b5470722444c1a0034a6b92f4cf6b"
format_stamp: "Formatted at 2026-09-24 16:15:43 on dist-test-slave-6wvm"
server_key: "a0ec5aabeb9f1926b361ed0fbe0e75d6"
server_key_iv: "8eb53001220a095b2127ba30baaff06c"
server_key_version: "encryptionkey@0"
I20260924 16:15:43.590698 19288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:43.613356 19288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:43.616714 19288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:43.616950 19288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:43.617625 19288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:43.618584 19288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:43.618659 19288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:43.618731 19288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:43.618764 19288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:43.627869 19288 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:42621
I20260924 16:15:43.627898 19419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:42621 every 8 connection(s)
I20260924 16:15:43.628775 19288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:43.631142 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 19288
I20260924 16:15:43.631296 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:15:43.631605 2037 external_mini_cluster.cc:1428] Setting key 8ac67081c1b5330c994bc72594245ffc
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
Sep 24 16:15:43 dist-test-slave-6wvm krb5kdc[19180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266543, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:15:43.653751 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.630753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:48665 (local address 127.1.253.126:41305)
0924 16:15:43.630938 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:43.630943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:43.631920 (+ 977us) server_negotiation.cc:407] Connection header received
0924 16:15:43.633460 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:43.633464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:43.633516 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:43.633608 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:43.634775 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.635468 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.637022 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.637470 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.637596 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:43.649367 (+ 11771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:43.649373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:43.649376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:43.649417 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:43.651275 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:43.651883 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:43.651890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:43.651893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:43.652023 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:43.652407 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:43.652410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:43.652414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:43.652700 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:43.652818 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:43.653359 (+ 541us) server_negotiation.cc:299] Negotiation successful
0924 16:15:43.653520 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:15:43.654627 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.631175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:41305 (local address 127.1.253.65:48665)
0924 16:15:43.631740 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:43.631795 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:43.633245 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:43.633737 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:43.633743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:43.634107 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:43.634623 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:43.634637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.635618 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.635623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:43.636880 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:43.636891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.637628 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.637631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:43.637826 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:43.646989 (+ 9163us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:43.647024 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:43.649096 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:43.651428 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:43.651436 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:43.651452 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:43.651751 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:43.652133 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:43.652136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:43.652137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:43.652291 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:43.652814 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:43.652820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:43.653087 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 16:15:43.653257 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 16:15:43.653543 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":183,"threads_started":1}
I20260924 16:15:43.656250 19420 heartbeater.cc:347] Connected to a master server at 127.1.253.126:41305
I20260924 16:15:43.656565 19420 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:43.657136 19420 heartbeater.cc:511] Master 127.1.253.126:41305 requested a full tablet report, sending...
I20260924 16:15:43.659067 19225 ts_manager.cc:194] Registered new tserver with Master: d97b5470722444c1a0034a6b92f4cf6b (127.1.253.65:42621)
I20260924 16:15:43.660511 19225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:48665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:15:43.666226 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41305
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:43.784034 19425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:43.784332 19425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:43.784410 19425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:43.788723 19425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:43.788810 19425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:43.788900 19425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:15:43.795287 19425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:43.796442 19425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:43.797538 19425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:43.808414 19430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:43.808414 19431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:43.808987 19433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:43.810954 19425 server_base.cc:1034] running on GCE node
I20260924 16:15:43.811564 19425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:43.812273 19425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:43.813476 19425 hybrid_clock.cc:648] HybridClock initialized: now 1790266543813456 us; error 30 us; skew 500 ppm
Sep 24 16:15:43 dist-test-slave-6wvm krb5kdc[19180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266543, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:43.816501 19425 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:15:43.817480 19425 webserver.cc:460] Webserver started at http://127.1.253.66:35399/ using document root <none> and password file <none>
I20260924 16:15:43.818070 19425 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:43.818181 19425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:43.818447 19425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:43.820565 19425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "6274281c3f9d4272921dc86d9313eba3"
format_stamp: "Formatted at 2026-09-24 16:15:43 on dist-test-slave-6wvm"
server_key: "61d132c522e1b1a8f7f1274bbdafc9cf"
server_key_iv: "74a0e26e772c263b804c8b2b87f3865c"
server_key_version: "encryptionkey@0"
I20260924 16:15:43.821172 19425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "6274281c3f9d4272921dc86d9313eba3"
format_stamp: "Formatted at 2026-09-24 16:15:43 on dist-test-slave-6wvm"
server_key: "61d132c522e1b1a8f7f1274bbdafc9cf"
server_key_iv: "74a0e26e772c263b804c8b2b87f3865c"
server_key_version: "encryptionkey@0"
I20260924 16:15:43.824987 19425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:15:43.827452 19440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:43.828639 19425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:15:43.828780 19425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "6274281c3f9d4272921dc86d9313eba3"
format_stamp: "Formatted at 2026-09-24 16:15:43 on dist-test-slave-6wvm"
server_key: "61d132c522e1b1a8f7f1274bbdafc9cf"
server_key_iv: "74a0e26e772c263b804c8b2b87f3865c"
server_key_version: "encryptionkey@0"
I20260924 16:15:43.828892 19425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:43.851367 19425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:43.854696 19425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:43.854923 19425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:43.855546 19425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:43.856428 19425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:43.856474 19425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:43.856551 19425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:43.856587 19425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:43.865283 19425 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:41209
I20260924 16:15:43.865352 19556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:41209 every 8 connection(s)
I20260924 16:15:43.866271 19425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:43.873509 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 19425
I20260924 16:15:43.873656 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:15:43.873998 2037 external_mini_cluster.cc:1428] Setting key 4bfb18ef08cb9b82dddb0d619785e3e5
Sep 24 16:15:43 dist-test-slave-6wvm krb5kdc[19180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266543, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:15:43.881067 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.868168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:34937 (local address 127.1.253.126:41305)
0924 16:15:43.868315 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:43.868319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:43.869134 (+ 815us) server_negotiation.cc:407] Connection header received
0924 16:15:43.870537 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:43.870541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:43.870606 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:43.870696 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:43.871950 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.872491 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.874016 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.874361 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.874457 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:43.877224 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:43.877228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:43.877230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:43.877254 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:43.878785 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:43.879370 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:43.879374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:43.879376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:43.879463 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:43.879858 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:43.879862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:43.879864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:43.880068 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:43.880193 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:43.880771 (+ 578us) server_negotiation.cc:299] Negotiation successful
0924 16:15:43.880898 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:15:43.881949 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.868508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:41305 (local address 127.1.253.66:34937)
0924 16:15:43.868971 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:43.869015 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:43.870302 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:43.870903 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:43.870911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:43.871278 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:43.871787 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:43.871801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.872647 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.872651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:43.873873 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:43.873887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.874548 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.874552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:43.874708 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:43.875370 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:43.875388 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:43.877028 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:43.878943 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:43.878951 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:43.878969 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:43.879250 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:43.879566 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:43.879569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:43.879571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:43.879747 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:43.880190 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:43.880198 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:43.880480 (+ 282us) client_negotiation.cc:769] Sending connection context
0924 16:15:43.880715 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 16:15:43.881082 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":127,"threads_started":1}
I20260924 16:15:43.883280 19557 heartbeater.cc:347] Connected to a master server at 127.1.253.126:41305
I20260924 16:15:43.883581 19557 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:43.884248 19557 heartbeater.cc:511] Master 127.1.253.126:41305 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:15:43.885524 19225 ts_manager.cc:194] Registered new tserver with Master: 6274281c3f9d4272921dc86d9313eba3 (127.1.253.66:41209)
I20260924 16:15:43.886134 19225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:34937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:15:43.905270 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41305
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:44.022377 19562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:44.022631 19562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:44.022675 19562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:44.026655 19562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:44.026708 19562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:44.026764 19562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:15:44.031894 19562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:44.032994 19562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:44.034030 19562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:44.044333 19568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:44.044344 19562 server_base.cc:1034] running on GCE node
W20260924 16:15:44.044344 19570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:44.044381 19567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:44.045146 19562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:44.045804 19562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:44.046989 19562 hybrid_clock.cc:648] HybridClock initialized: now 1790266544046954 us; error 47 us; skew 500 ppm
Sep 24 16:15:44 dist-test-slave-6wvm krb5kdc[19180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266544, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:44.049806 19562 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:15:44.050768 19562 webserver.cc:460] Webserver started at http://127.1.253.67:39221/ using document root <none> and password file <none>
I20260924 16:15:44.051375 19562 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:44.051468 19562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:44.051654 19562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:44.053674 19562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "15bd04080c254df4a60226d8f2fc695d"
format_stamp: "Formatted at 2026-09-24 16:15:44 on dist-test-slave-6wvm"
server_key: "cb433223f967c625d5de332112905d81"
server_key_iv: "c855b93d0c46ac1a94e5902f8a5b1304"
server_key_version: "encryptionkey@0"
I20260924 16:15:44.054193 19562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "15bd04080c254df4a60226d8f2fc695d"
format_stamp: "Formatted at 2026-09-24 16:15:44 on dist-test-slave-6wvm"
server_key: "cb433223f967c625d5de332112905d81"
server_key_iv: "c855b93d0c46ac1a94e5902f8a5b1304"
server_key_version: "encryptionkey@0"
I20260924 16:15:44.058048 19562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 16:15:44.060724 19577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:44.062103 19562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:15:44.062307 19562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "15bd04080c254df4a60226d8f2fc695d"
format_stamp: "Formatted at 2026-09-24 16:15:44 on dist-test-slave-6wvm"
server_key: "cb433223f967c625d5de332112905d81"
server_key_iv: "c855b93d0c46ac1a94e5902f8a5b1304"
server_key_version: "encryptionkey@0"
I20260924 16:15:44.062438 19562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:44.079171 19562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:44.082535 19562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:44.082758 19562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:44.083597 19562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:44.084877 19562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:44.084950 19562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:44.085016 19562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:44.085054 19562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:44.095721 19562 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:36015
I20260924 16:15:44.095741 19691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:36015 every 8 connection(s)
I20260924 16:15:44.096688 19562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:44.102492 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 19562
I20260924 16:15:44.102686 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:15:44.103020 2037 external_mini_cluster.cc:1428] Setting key e1691809d34dec0ffff4190b38ba77ab
Sep 24 16:15:44 dist-test-slave-6wvm krb5kdc[19180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266544, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:15:44.111899 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.098704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:32971 (local address 127.1.253.126:41305)
0924 16:15:44.098885 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:44.098889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:44.099816 (+ 927us) server_negotiation.cc:407] Connection header received
0924 16:15:44.101183 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:44.101186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:44.101238 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:44.101318 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:44.102768 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.103502 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.104880 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.105110 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.105239 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.108181 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:44.108185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:44.108187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:44.108216 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:44.109667 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:44.110198 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:44.110201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:44.110202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:44.110362 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:44.110693 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:44.110696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:44.110697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:44.110887 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:44.110976 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:44.111631 (+ 655us) server_negotiation.cc:299] Negotiation successful
0924 16:15:44.111754 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:15:44.112775 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.099135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:41305 (local address 127.1.253.67:32971)
0924 16:15:44.099649 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:44.099694 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:44.100964 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:44.101494 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:44.101500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:44.101903 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:44.102570 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.102592 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.103656 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.103660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.104738 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.104748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.105296 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.105299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.105508 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.106189 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:44.106206 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:44.107980 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:44.109795 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:44.109800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:44.109814 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:44.110097 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:44.110455 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:44.110457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:44.110459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:44.110602 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:44.111094 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:44.111101 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:44.111362 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 16:15:44.111544 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 16:15:44.111840 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":118,"threads_started":1}
I20260924 16:15:44.114300 19692 heartbeater.cc:347] Connected to a master server at 127.1.253.126:41305
I20260924 16:15:44.114611 19692 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:44.115180 19692 heartbeater.cc:511] Master 127.1.253.126:41305 requested a full tablet report, sending...
I20260924 16:15:44.116614 19221 ts_manager.cc:194] Registered new tserver with Master: 15bd04080c254df4a60226d8f2fc695d (127.1.253.67:36015)
I20260924 16:15:44.117347 19221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:32971
I20260924 16:15:44.117652 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:15:44.127604 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.120049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.1.253.126:41305)
0924 16:15:44.120226 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:44.120231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:44.120385 (+ 154us) server_negotiation.cc:407] Connection header received
0924 16:15:44.120553 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:44.120557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:44.120625 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:44.120718 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:44.121453 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.121964 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.122907 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.123145 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.123262 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.124601 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:44.124605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:44.124608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:44.124641 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:44.125841 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:44.126303 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:44.126306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:44.126308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:44.126393 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:44.126694 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:44.126698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:44.126699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:44.126915 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:44.127069 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:44.127320 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 16:15:44.127439 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:15:44.130915 19221 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:15:44.133939 19221 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:15:44.149166 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.143697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.1.253.67:36015)
0924 16:15:44.144524 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:44.144537 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:44.144559 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:15:44.144634 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:44.144641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:44.144818 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:44.144960 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:44.145252 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.146056 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.147688 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.148347 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.148460 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.148618 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:44.148893 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 16:15:44.148993 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":176,"threads_started":1}
I20260924 16:15:44.149165 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.143085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:36015 (local address 127.0.0.1:34708)
0924 16:15:44.143850 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:44.143863 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:44.143952 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:44.144949 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:44.144952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:44.144971 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:44.145086 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.145093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.146194 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.146201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.147549 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.147559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.148505 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.148510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.148665 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.148711 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 16:15:44.148839 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 16:15:44.148969 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":64,"thread_start_us":103,"threads_started":1}
I20260924 16:15:44.149513 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.142936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:42621 (local address 127.0.0.1:60234)
0924 16:15:44.143402 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:44.143447 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:44.143572 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:44.144762 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:44.144769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:44.144792 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:44.144929 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.144937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.146595 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.146599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.148010 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.148022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.149107 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.149111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.149234 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.149265 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:15:44.149329 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 16:15:44.149396 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":3,"thread_start_us":120,"threads_started":1}
I20260924 16:15:44.150149 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.143830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:41209 (local address 127.0.0.1:52250)
0924 16:15:44.144188 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:44.144204 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:44.144299 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:44.145193 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:44.145195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:44.145216 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:44.145314 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.145320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.146804 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.146808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.148048 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.148060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.149199 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.149204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.149812 (+ 608us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.149838 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 16:15:44.149917 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 16:15:44.149986 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":471,"thread_start_us":108,"threads_started":1}
I20260924 16:15:44.150813 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.143055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.1.253.65:42621)
0924 16:15:44.144198 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:44.144219 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:44.144244 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:15:44.144326 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:44.144331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:44.144553 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:44.144754 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:44.145088 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.146441 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.148158 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.148958 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.150376 (+ 1418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.150528 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:44.150577 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 16:15:44.150671 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":127,"threads_started":1}
I20260924 16:15:44.152719 19626 tablet_service.cc:1467] Processing CreateTablet for tablet 6c849d75452842faa9fda805ef03dd45 (DEFAULT_TABLE table=test-table [id=15678fd02bcf47b2beb928e0d493640e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:44.153290 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.144276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.1.253.66:41209)
0924 16:15:44.144643 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:44.144660 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:44.144687 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:15:44.144771 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:44.144776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:44.144992 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:44.145197 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:44.145451 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.146651 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.148207 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.149053 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.149266 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.149422 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:44.149981 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 16:15:44.150076 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":173,"threads_started":1}
I20260924 16:15:44.153272 19353 tablet_service.cc:1467] Processing CreateTablet for tablet 6c849d75452842faa9fda805ef03dd45 (DEFAULT_TABLE table=test-table [id=15678fd02bcf47b2beb928e0d493640e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:44.153939 19626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c849d75452842faa9fda805ef03dd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:44.154465 19353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c849d75452842faa9fda805ef03dd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:44.155936 19489 tablet_service.cc:1467] Processing CreateTablet for tablet 6c849d75452842faa9fda805ef03dd45 (DEFAULT_TABLE table=test-table [id=15678fd02bcf47b2beb928e0d493640e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:44.156822 19489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c849d75452842faa9fda805ef03dd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:44.162034 19709 tablet_bootstrap.cc:492] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b: Bootstrap starting.
I20260924 16:15:44.162911 19710 tablet_bootstrap.cc:492] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d: Bootstrap starting.
I20260924 16:15:44.163826 19711 tablet_bootstrap.cc:492] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3: Bootstrap starting.
I20260924 16:15:44.165338 19709 tablet_bootstrap.cc:654] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:44.165848 19710 tablet_bootstrap.cc:654] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:44.166311 19711 tablet_bootstrap.cc:654] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:44.166395 19709 log.cc:824] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:44.167150 19711 log.cc:824] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:44.167300 19710 log.cc:824] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:44.168730 19709 tablet_bootstrap.cc:492] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b: No bootstrap required, opened a new log
I20260924 16:15:44.168989 19709 ts_tablet_manager.cc:1397] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260924 16:15:44.169142 19710 tablet_bootstrap.cc:492] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d: No bootstrap required, opened a new log
I20260924 16:15:44.169257 19711 tablet_bootstrap.cc:492] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3: No bootstrap required, opened a new log
I20260924 16:15:44.169361 19710 ts_tablet_manager.cc:1397] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260924 16:15:44.169463 19711 ts_tablet_manager.cc:1397] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:15:44.172235 19710 raft_consensus.cc:348] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97b5470722444c1a0034a6b92f4cf6b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42621 } } peers { permanent_uuid: "15bd04080c254df4a60226d8f2fc695d" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36015 } } peers { permanent_uuid: "6274281c3f9d4272921dc86d9313eba3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41209 } }
I20260924 16:15:44.172501 19710 raft_consensus.cc:374] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:44.172564 19710 raft_consensus.cc:729] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bd04080c254df4a60226d8f2fc695d, State: Initialized, Role: FOLLOWER
I20260924 16:15:44.173046 19710 consensus_queue.cc:260] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97b5470722444c1a0034a6b92f4cf6b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42621 } } peers { permanent_uuid: "15bd04080c254df4a60226d8f2fc695d" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36015 } } peers { permanent_uuid: "6274281c3f9d4272921dc86d9313eba3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41209 } }
I20260924 16:15:44.173722 19692 heartbeater.cc:503] Master 127.1.253.126:41305 was elected leader, sending a full tablet report...
I20260924 16:15:44.173702 19711 raft_consensus.cc:348] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97b5470722444c1a0034a6b92f4cf6b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42621 } } peers { permanent_uuid: "15bd04080c254df4a60226d8f2fc695d" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36015 } } peers { permanent_uuid: "6274281c3f9d4272921dc86d9313eba3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41209 } }
I20260924 16:15:44.173961 19711 raft_consensus.cc:374] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:44.174011 19711 raft_consensus.cc:729] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6274281c3f9d4272921dc86d9313eba3, State: Initialized, Role: FOLLOWER
I20260924 16:15:44.174103 19710 ts_tablet_manager.cc:1428] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260924 16:15:44.174484 19711 consensus_queue.cc:260] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97b5470722444c1a0034a6b92f4cf6b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42621 } } peers { permanent_uuid: "15bd04080c254df4a60226d8f2fc695d" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36015 } } peers { permanent_uuid: "6274281c3f9d4272921dc86d9313eba3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41209 } }
I20260924 16:15:44.174830 19709 raft_consensus.cc:348] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97b5470722444c1a0034a6b92f4cf6b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42621 } } peers { permanent_uuid: "15bd04080c254df4a60226d8f2fc695d" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36015 } } peers { permanent_uuid: "6274281c3f9d4272921dc86d9313eba3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41209 } }
I20260924 16:15:44.175084 19709 raft_consensus.cc:374] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:44.175138 19709 raft_consensus.cc:729] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97b5470722444c1a0034a6b92f4cf6b, State: Initialized, Role: FOLLOWER
I20260924 16:15:44.175243 19557 heartbeater.cc:503] Master 127.1.253.126:41305 was elected leader, sending a full tablet report...
I20260924 16:15:44.175511 19711 ts_tablet_manager.cc:1428] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 16:15:44.175596 19709 consensus_queue.cc:260] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97b5470722444c1a0034a6b92f4cf6b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42621 } } peers { permanent_uuid: "15bd04080c254df4a60226d8f2fc695d" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36015 } } peers { permanent_uuid: "6274281c3f9d4272921dc86d9313eba3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41209 } }
I20260924 16:15:44.176301 19420 heartbeater.cc:503] Master 127.1.253.126:41305 was elected leader, sending a full tablet report...
I20260924 16:15:44.176635 19709 ts_tablet_manager.cc:1428] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 16:15:44.251116 19716 raft_consensus.cc:482] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:44.251351 19716 raft_consensus.cc:504] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97b5470722444c1a0034a6b92f4cf6b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42621 } } peers { permanent_uuid: "15bd04080c254df4a60226d8f2fc695d" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36015 } } peers { permanent_uuid: "6274281c3f9d4272921dc86d9313eba3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41209 } }
I20260924 16:15:44.252573 19716 leader_election.cc:288] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d97b5470722444c1a0034a6b92f4cf6b (127.1.253.65:42621), 15bd04080c254df4a60226d8f2fc695d (127.1.253.67:36015)
I20260924 16:15:44.257476 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.253012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:39005 (local address 127.1.253.65:42621)
0924 16:15:44.253189 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:44.253195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:44.253212 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:15:44.253273 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:44.253277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:44.253337 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:44.253429 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:44.253817 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.254612 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.256136 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.256773 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.256906 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.256977 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:44.257265 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 16:15:44.257338 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:15:44.257476 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.253011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:48573 (local address 127.1.253.67:36015)
0924 16:15:44.253160 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:44.253164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:44.253779 (+ 615us) server_negotiation.cc:407] Connection header received
0924 16:15:44.253843 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:44.253846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:44.253897 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:44.253977 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:44.254262 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.254904 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.256264 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.256872 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.256984 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.257040 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:44.257264 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 16:15:44.257338 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:15:44.257537 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.252875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:42621 (local address 127.1.253.66:39005)
0924 16:15:44.253000 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:44.253017 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:44.253122 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:44.253509 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:44.253512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:44.253540 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:44.253678 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.253688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.254723 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.254726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.255993 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.256004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.256994 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.256998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.257116 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.257137 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:15:44.257228 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:15:44.257353 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":41}
I20260924 16:15:44.257550 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.252875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:36015 (local address 127.1.253.66:48573)
0924 16:15:44.253637 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:44.253652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:44.253769 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:44.254003 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:44.254006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:44.254023 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:44.254126 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.254131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.255019 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.255023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.256153 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.256160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.257006 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.257009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.257110 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.257140 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:15:44.257228 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 16:15:44.257353 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":345,"thread_start_us":160,"threads_started":1}
I20260924 16:15:44.258342 19374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c849d75452842faa9fda805ef03dd45" candidate_uuid: "6274281c3f9d4272921dc86d9313eba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97b5470722444c1a0034a6b92f4cf6b" is_pre_election: true
I20260924 16:15:44.258481 19646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c849d75452842faa9fda805ef03dd45" candidate_uuid: "6274281c3f9d4272921dc86d9313eba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bd04080c254df4a60226d8f2fc695d" is_pre_election: true
I20260924 16:15:44.258736 19374 raft_consensus.cc:2454] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6274281c3f9d4272921dc86d9313eba3 in term 0.
I20260924 16:15:44.258821 19646 raft_consensus.cc:2454] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6274281c3f9d4272921dc86d9313eba3 in term 0.
I20260924 16:15:44.259356 19441 leader_election.cc:302] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 15bd04080c254df4a60226d8f2fc695d, 6274281c3f9d4272921dc86d9313eba3, d97b5470722444c1a0034a6b92f4cf6b; no voters:
I20260924 16:15:44.259657 19716 raft_consensus.cc:2781] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:15:44.259752 19716 raft_consensus.cc:482] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:15:44.259786 19716 raft_consensus.cc:3037] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:44.260898 19716 raft_consensus.cc:504] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97b5470722444c1a0034a6b92f4cf6b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42621 } } peers { permanent_uuid: "15bd04080c254df4a60226d8f2fc695d" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36015 } } peers { permanent_uuid: "6274281c3f9d4272921dc86d9313eba3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41209 } }
I20260924 16:15:44.261303 19716 leader_election.cc:288] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [CANDIDATE]: Term 1 election: Requested vote from peers d97b5470722444c1a0034a6b92f4cf6b (127.1.253.65:42621), 15bd04080c254df4a60226d8f2fc695d (127.1.253.67:36015)
I20260924 16:15:44.261674 19374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c849d75452842faa9fda805ef03dd45" candidate_uuid: "6274281c3f9d4272921dc86d9313eba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97b5470722444c1a0034a6b92f4cf6b"
I20260924 16:15:44.261693 19646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c849d75452842faa9fda805ef03dd45" candidate_uuid: "6274281c3f9d4272921dc86d9313eba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bd04080c254df4a60226d8f2fc695d"
I20260924 16:15:44.261826 19374 raft_consensus.cc:3037] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:44.261842 19646 raft_consensus.cc:3037] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:44.263250 19374 raft_consensus.cc:2454] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6274281c3f9d4272921dc86d9313eba3 in term 1.
I20260924 16:15:44.263250 19646 raft_consensus.cc:2454] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6274281c3f9d4272921dc86d9313eba3 in term 1.
I20260924 16:15:44.263593 19444 leader_election.cc:302] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15bd04080c254df4a60226d8f2fc695d, 6274281c3f9d4272921dc86d9313eba3; no voters:
I20260924 16:15:44.263782 19716 raft_consensus.cc:2781] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:44.263998 19716 raft_consensus.cc:686] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 1 LEADER]: Becoming Leader. State: Replica: 6274281c3f9d4272921dc86d9313eba3, State: Running, Role: LEADER
I20260924 16:15:44.264369 19716 consensus_queue.cc:237] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97b5470722444c1a0034a6b92f4cf6b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42621 } } peers { permanent_uuid: "15bd04080c254df4a60226d8f2fc695d" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36015 } } peers { permanent_uuid: "6274281c3f9d4272921dc86d9313eba3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41209 } }
I20260924 16:15:44.267822 19221 catalog_manager.cc:5495] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6274281c3f9d4272921dc86d9313eba3 (127.1.253.66). New cstate: current_term: 1 leader_uuid: "6274281c3f9d4272921dc86d9313eba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97b5470722444c1a0034a6b92f4cf6b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15bd04080c254df4a60226d8f2fc695d" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6274281c3f9d4272921dc86d9313eba3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 41209 } health_report { overall_health: HEALTHY } } }
I20260924 16:15:44.302405 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.298544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.1.253.66:41209)
0924 16:15:44.298710 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:44.298716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:44.298732 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:15:44.298836 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:44.298839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:44.298909 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:44.299033 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:15:44.299312 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.299834 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.300755 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.300997 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.301123 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.301471 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:15:44.301918 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:15:44.302084 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:44.302146 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 16:15:44.302206 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:15:44.308470 19646 raft_consensus.cc:1261] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d [term 1 FOLLOWER]: Refusing update from remote peer 6274281c3f9d4272921dc86d9313eba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:44.308526 19374 raft_consensus.cc:1261] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b [term 1 FOLLOWER]: Refusing update from remote peer 6274281c3f9d4272921dc86d9313eba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:44.309317 19719 consensus_queue.cc:1035] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97b5470722444c1a0034a6b92f4cf6b" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:44.309507 19716 consensus_queue.cc:1035] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15bd04080c254df4a60226d8f2fc695d" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:44.320894 19722 mvcc.cc:205] Tried to move back new op lower bound from 7332931765480022016 to 7332931765310062592. Current Snapshot: MvccSnapshot[applied={T|T < 7332931765480022016}]
I20260924 16:15:44.321516 19723 mvcc.cc:205] Tried to move back new op lower bound from 7332931765480022016 to 7332931765310062592. Current Snapshot: MvccSnapshot[applied={T|T < 7332931765480022016 or (T in {7332931765480022016})}]
I20260924 16:15:44.330597 19225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:44.330785 19225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:44.333637 19225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cb8b9239ca748f485dafe5fe3533a22: Sending DeleteTablet for 3 replicas of tablet 6c849d75452842faa9fda805ef03dd45
I20260924 16:15:44.334506 19626 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c849d75452842faa9fda805ef03dd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:34708
I20260924 16:15:44.334493 19353 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c849d75452842faa9fda805ef03dd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:60234
I20260924 16:15:44.335008 19732 tablet_replica.cc:330] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b: stopping tablet replica
I20260924 16:15:44.335026 19489 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c849d75452842faa9fda805ef03dd45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:44 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:52250
I20260924 16:15:44.335350 19732 raft_consensus.cc:2229] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:44.335596 19733 tablet_replica.cc:330] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d: stopping tablet replica
I20260924 16:15:44.335722 19732 raft_consensus.cc:2258] T 6c849d75452842faa9fda805ef03dd45 P d97b5470722444c1a0034a6b92f4cf6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:44.335928 19733 raft_consensus.cc:2229] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:44.336063 19734 tablet_replica.cc:330] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3: stopping tablet replica
I20260924 16:15:44.336236 19733 raft_consensus.cc:2258] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:44.336315 19734 raft_consensus.cc:2229] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:15:44.336567 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 19288
I20260924 16:15:44.337347 19734 raft_consensus.cc:2258] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:44.339247 19734 ts_tablet_manager.cc:1905] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:44.342861 19734 ts_tablet_manager.cc:1918] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:44.342969 19734 log.cc:1196] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/6c849d75452842faa9fda805ef03dd45
I20260924 16:15:44.343284 19734 ts_tablet_manager.cc:1939] T 6c849d75452842faa9fda805ef03dd45 P 6274281c3f9d4272921dc86d9313eba3: Deleting consensus metadata
I20260924 16:15:44.344370 19733 ts_tablet_manager.cc:1905] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:44.344592 19211 catalog_manager.cc:4841] TS 6274281c3f9d4272921dc86d9313eba3 (127.1.253.66:41209): tablet 6c849d75452842faa9fda805ef03dd45 (table test-table [id=15678fd02bcf47b2beb928e0d493640e]) successfully deleted
I20260924 16:15:44.347234 19733 ts_tablet_manager.cc:1918] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 16:15:44.347236 19209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:42621 (error 108)
I20260924 16:15:44.347342 19733 log.cc:1196] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/6c849d75452842faa9fda805ef03dd45
I20260924 16:15:44.347677 19733 ts_tablet_manager.cc:1939] T 6c849d75452842faa9fda805ef03dd45 P 15bd04080c254df4a60226d8f2fc695d: Deleting consensus metadata
I20260924 16:15:44.348415 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.348077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:42621 (local address 127.0.0.1:60248)
0924 16:15:44.348219 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:44.348296 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 16:15:44.348495 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 19425
W20260924 16:15:44.349752 19209 catalog_manager.cc:4584] TS d97b5470722444c1a0034a6b92f4cf6b (127.1.253.65:42621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c849d75452842faa9fda805ef03dd45: Network error: Client connection negotiation failed: client connection to 127.1.253.65:42621: connect: Connection refused (error 111)
I20260924 16:15:44.350597 19212 catalog_manager.cc:4841] TS 15bd04080c254df4a60226d8f2fc695d (127.1.253.67:36015): tablet 6c849d75452842faa9fda805ef03dd45 (table test-table [id=15678fd02bcf47b2beb928e0d493640e]) successfully deleted
I20260924 16:15:44.355618 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 19562
I20260924 16:15:44.362226 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 19193
2026-09-24T16:15:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2425 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T16:15:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:44Z Disabled control of system clock
I20260924 16:15:44.395329 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:41731
--webserver_interface=127.1.253.126
--webserver_port=0
--builtin_ntp_servers=127.1.253.84:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:41731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:15:44.508452 19739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:44.508735 19739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:44.508771 19739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:44.512957 19739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:44.513021 19739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:44.513037 19739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:44.513054 19739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:44.513069 19739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:44.518147 19739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:41731
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:41731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:44.519436 19739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:44.521098 19739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:44.530212 19747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:44.530212 19744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:44.530354 19739 server_base.cc:1034] running on GCE node
W20260924 16:15:44.530184 19745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:44.531261 19739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:44.532371 19739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:44.533612 19739 hybrid_clock.cc:648] HybridClock initialized: now 1790266544533594 us; error 42 us; skew 500 ppm
I20260924 16:15:44.535845 19739 webserver.cc:460] Webserver started at http://127.1.253.126:40653/ using document root <none> and password file <none>
I20260924 16:15:44.536522 19739 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:44.536634 19739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:44.536895 19739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:44.539218 19739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "fe3726a3893742f5b3132b137b5c5fc0"
format_stamp: "Formatted at 2026-09-24 16:15:44 on dist-test-slave-6wvm"
server_key: "abede71b3d85ff51b379bb0cef78ac42"
server_key_iv: "811023da334d71356926422b309326d7"
server_key_version: "encryptionkey@0"
I20260924 16:15:44.539832 19739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "fe3726a3893742f5b3132b137b5c5fc0"
format_stamp: "Formatted at 2026-09-24 16:15:44 on dist-test-slave-6wvm"
server_key: "abede71b3d85ff51b379bb0cef78ac42"
server_key_iv: "811023da334d71356926422b309326d7"
server_key_version: "encryptionkey@0"
I20260924 16:15:44.544063 19739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:15:44.546911 19753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:44.548267 19739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:44.548444 19739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "fe3726a3893742f5b3132b137b5c5fc0"
format_stamp: "Formatted at 2026-09-24 16:15:44 on dist-test-slave-6wvm"
server_key: "abede71b3d85ff51b379bb0cef78ac42"
server_key_iv: "811023da334d71356926422b309326d7"
server_key_version: "encryptionkey@0"
I20260924 16:15:44.548578 19739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:44.565826 19739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:44.566689 19739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:44.566911 19739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:44.574400 19739 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:41731
I20260924 16:15:44.574409 19806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:41731 every 8 connection(s)
I20260924 16:15:44.575420 19739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:44.578596 19807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:44.581759 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 19739
I20260924 16:15:44.581868 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:15:44.582196 2037 external_mini_cluster.cc:1428] Setting key 81c7cd3117afd57b99539126c5528668
I20260924 16:15:44.584645 19807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0: Bootstrap starting.
I20260924 16:15:44.587347 19807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:44.588186 19807 log.cc:824] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:44.590401 19807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0: No bootstrap required, opened a new log
I20260924 16:15:44.593255 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.583813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59766 (local address 127.1.253.126:41731)
0924 16:15:44.584503 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:44.584525 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:44.584566 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:15:44.585408 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:44.585433 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:44.585946 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:44.586345 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:44.586793 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.588254 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.589427 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.590100 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.590285 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.591197 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:44.591203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:44.591219 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:44.591711 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:44.591781 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:44.591807 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:44.591908 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:44.592014 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 16:15:44.592315 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":104,"threads_started":1}
I20260924 16:15:44.594795 19807 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3726a3893742f5b3132b137b5c5fc0" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41731 } }
I20260924 16:15:44.595012 19807 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:44.595053 19807 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3726a3893742f5b3132b137b5c5fc0, State: Initialized, Role: FOLLOWER
I20260924 16:15:44.595526 19807 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3726a3893742f5b3132b137b5c5fc0" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41731 } }
I20260924 16:15:44.595700 19807 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:44.595770 19807 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:44.595870 19807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:44.596827 19807 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3726a3893742f5b3132b137b5c5fc0" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41731 } }
I20260924 16:15:44.597182 19807 leader_election.cc:302] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe3726a3893742f5b3132b137b5c5fc0; no voters:
I20260924 16:15:44.597501 19807 leader_election.cc:288] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:44.597666 19812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:44.597906 19812 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [term 1 LEADER]: Becoming Leader. State: Replica: fe3726a3893742f5b3132b137b5c5fc0, State: Running, Role: LEADER
I20260924 16:15:44.598217 19812 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3726a3893742f5b3132b137b5c5fc0" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41731 } }
I20260924 16:15:44.598832 19807 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:44.599972 19813 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe3726a3893742f5b3132b137b5c5fc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3726a3893742f5b3132b137b5c5fc0" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41731 } } }
I20260924 16:15:44.600111 19813 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:44.600402 19814 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe3726a3893742f5b3132b137b5c5fc0. Latest consensus state: current_term: 1 leader_uuid: "fe3726a3893742f5b3132b137b5c5fc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3726a3893742f5b3132b137b5c5fc0" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41731 } } }
I20260924 16:15:44.600498 19814 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:44.600651 19821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:44.603865 19821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:44.610020 19821 catalog_manager.cc:1347] Generated new cluster ID: 65f40dc7b981417c931b08556c3cb5d3
I20260924 16:15:44.610096 19821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:44.631815 19821 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:44.633078 19821 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:44.649824 19821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0: Generated new TSK 0
I20260924 16:15:44.650615 19821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:15:44.658108 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41731
--builtin_ntp_servers=127.1.253.84:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:15:44.775501 19831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:44.775776 19831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:44.775816 19831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:44.779937 19831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:44.780000 19831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:44.780057 19831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:15:44.784952 19831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:44.786259 19831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:44.788100 19831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:44.798421 19836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:44.798439 19839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:44.798632 19831 server_base.cc:1034] running on GCE node
W20260924 16:15:44.798450 19837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:44.799046 19831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:44.799784 19831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:44.800940 19831 hybrid_clock.cc:648] HybridClock initialized: now 1790266544800904 us; error 49 us; skew 500 ppm
I20260924 16:15:44.802848 19831 webserver.cc:460] Webserver started at http://127.1.253.65:46233/ using document root <none> and password file <none>
I20260924 16:15:44.803443 19831 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:44.803505 19831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:44.803738 19831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:44.805838 19831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "58494fdc54074f879557c1cc4c97f30a"
format_stamp: "Formatted at 2026-09-24 16:15:44 on dist-test-slave-6wvm"
server_key: "0fa9ea78d3b9133548f64216805654fe"
server_key_iv: "b7389529ce65afd703c2992985d811dd"
server_key_version: "encryptionkey@0"
I20260924 16:15:44.806452 19831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "58494fdc54074f879557c1cc4c97f30a"
format_stamp: "Formatted at 2026-09-24 16:15:44 on dist-test-slave-6wvm"
server_key: "0fa9ea78d3b9133548f64216805654fe"
server_key_iv: "b7389529ce65afd703c2992985d811dd"
server_key_version: "encryptionkey@0"
I20260924 16:15:44.810135 19831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:15:44.812666 19845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:44.813946 19831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:15:44.814050 19831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "58494fdc54074f879557c1cc4c97f30a"
format_stamp: "Formatted at 2026-09-24 16:15:44 on dist-test-slave-6wvm"
server_key: "0fa9ea78d3b9133548f64216805654fe"
server_key_iv: "b7389529ce65afd703c2992985d811dd"
server_key_version: "encryptionkey@0"
I20260924 16:15:44.814150 19831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:44.852114 19831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:44.852897 19831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:44.853060 19831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:44.853683 19831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:44.854756 19831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:44.854806 19831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:44.854851 19831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:44.854866 19831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:44.863924 19831 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:33491
I20260924 16:15:44.863943 19960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:33491 every 8 connection(s)
I20260924 16:15:44.864879 19831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:44.866047 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 19831
I20260924 16:15:44.866192 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:15:44.866582 2037 external_mini_cluster.cc:1428] Setting key 2583c052f993391f62dc683caa7c7ed4
I20260924 16:15:44.868909 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41731
--builtin_ntp_servers=127.1.253.84:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:15:44.875854 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.867046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:38799 (local address 127.1.253.126:41731)
0924 16:15:44.867234 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:44.867239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:44.868656 (+ 1417us) server_negotiation.cc:407] Connection header received
0924 16:15:44.870783 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:44.870786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:44.870845 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:44.870926 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:44.871886 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.872461 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.873725 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.873964 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.874089 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.875310 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:44.875314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:44.875317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:44.875332 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:44.875361 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:44.875370 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:44.875464 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:44.875638 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 16:15:44.875707 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:15:44.876748 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.867563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:41731 (local address 127.1.253.65:38799)
0924 16:15:44.868481 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:44.868538 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:44.870526 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:44.871144 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:44.871152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:44.871241 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:44.871733 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.871745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.872609 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.872613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.873587 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:44.873598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:44.874134 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:44.874138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:44.874363 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:44.875032 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:15:44.875034 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:44.875077 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:15:44.875143 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:44.875468 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:44.875480 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:44.875486 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:15:44.875626 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 16:15:44.875927 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":137,"threads_started":1}
I20260924 16:15:44.878512 19961 heartbeater.cc:347] Connected to a master server at 127.1.253.126:41731
I20260924 16:15:44.878823 19961 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:44.879449 19961 heartbeater.cc:511] Master 127.1.253.126:41731 requested a full tablet report, sending...
I20260924 16:15:44.881451 19770 ts_manager.cc:194] Registered new tserver with Master: 58494fdc54074f879557c1cc4c97f30a (127.1.253.65:33491)
I20260924 16:15:44.883204 19770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.253.65:38799
W20260924 16:15:44.985970 19964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:44.986299 19964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:44.986362 19964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:44.990214 19964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:44.990332 19964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:44.990389 19964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:15:44.995112 19964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:44.996114 19964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:44.997494 19964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:45.007838 19970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:45.007977 19964 server_base.cc:1034] running on GCE node
W20260924 16:15:45.007845 19972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:45.007845 19969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:45.008647 19964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:45.009320 19964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:45.010500 19964 hybrid_clock.cc:648] HybridClock initialized: now 1790266545010473 us; error 40 us; skew 500 ppm
I20260924 16:15:45.012320 19964 webserver.cc:460] Webserver started at http://127.1.253.66:40797/ using document root <none> and password file <none>
I20260924 16:15:45.012979 19964 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:45.013041 19964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:45.013293 19964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:45.015470 19964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "3dfb921458e2468b8b47d5e33fcf4dc2"
format_stamp: "Formatted at 2026-09-24 16:15:45 on dist-test-slave-6wvm"
server_key: "aac48d7f455bbd7dc36a7539e5845507"
server_key_iv: "054c018a8c5a17aac49e4117b40e3c32"
server_key_version: "encryptionkey@0"
I20260924 16:15:45.016060 19964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "3dfb921458e2468b8b47d5e33fcf4dc2"
format_stamp: "Formatted at 2026-09-24 16:15:45 on dist-test-slave-6wvm"
server_key: "aac48d7f455bbd7dc36a7539e5845507"
server_key_iv: "054c018a8c5a17aac49e4117b40e3c32"
server_key_version: "encryptionkey@0"
I20260924 16:15:45.020509 19964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:15:45.023015 19978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:45.024289 19964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:15:45.024433 19964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "3dfb921458e2468b8b47d5e33fcf4dc2"
format_stamp: "Formatted at 2026-09-24 16:15:45 on dist-test-slave-6wvm"
server_key: "aac48d7f455bbd7dc36a7539e5845507"
server_key_iv: "054c018a8c5a17aac49e4117b40e3c32"
server_key_version: "encryptionkey@0"
I20260924 16:15:45.024539 19964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:45.038362 19964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:45.039049 19964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:45.039261 19964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:45.039870 19964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:45.040858 19964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:45.040905 19964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:45.040992 19964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:45.041036 19964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:45.049930 19964 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:43881
I20260924 16:15:45.049973 20094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:43881 every 8 connection(s)
I20260924 16:15:45.050858 19964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:45.056103 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 19964
I20260924 16:15:45.056269 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:15:45.056574 2037 external_mini_cluster.cc:1428] Setting key 80eea7556f719757e9405f13cfae7f2d
I20260924 16:15:45.058663 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41731
--builtin_ntp_servers=127.1.253.84:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:15:45.061025 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.052729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:42641 (local address 127.1.253.126:41731)
0924 16:15:45.052871 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:45.052874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:45.054086 (+ 1212us) server_negotiation.cc:407] Connection header received
0924 16:15:45.055578 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:45.055581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:45.055639 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:45.055764 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:45.056823 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.057481 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.058724 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.059040 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.059168 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.060464 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:45.060468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:45.060470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:45.060485 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:45.060509 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:45.060519 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:45.060628 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:45.060853 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 16:15:45.060912 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:15:45.062223 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.053099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:41731 (local address 127.1.253.66:42641)
0924 16:15:45.053889 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:45.053976 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:45.055353 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:45.055972 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:45.055978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:45.056065 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:45.056646 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.056683 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.057618 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.057621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.058593 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.058604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.059207 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.059212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.059379 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.060087 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:15:45.060090 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:45.060159 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:15:45.060253 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:45.060653 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:45.060663 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:45.060672 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:15:45.060817 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 16:15:45.061148 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260924 16:15:45.063513 20095 heartbeater.cc:347] Connected to a master server at 127.1.253.126:41731
I20260924 16:15:45.063828 20095 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:45.064479 20095 heartbeater.cc:511] Master 127.1.253.126:41731 requested a full tablet report, sending...
I20260924 16:15:45.065742 19770 ts_manager.cc:194] Registered new tserver with Master: 3dfb921458e2468b8b47d5e33fcf4dc2 (127.1.253.66:43881)
I20260924 16:15:45.066505 19770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.253.66:42641
W20260924 16:15:45.173205 20098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:45.173494 20098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:45.173583 20098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:45.177549 20098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:45.177637 20098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:45.177728 20098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:15:45.182480 20098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:45.183593 20098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:45.185017 20098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:45.194803 20103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:45.194803 20106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:45.194813 20104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:45.195048 20098 server_base.cc:1034] running on GCE node
I20260924 16:15:45.195538 20098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:45.196228 20098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:45.197424 20098 hybrid_clock.cc:648] HybridClock initialized: now 1790266545197406 us; error 29 us; skew 500 ppm
I20260924 16:15:45.199352 20098 webserver.cc:460] Webserver started at http://127.1.253.67:33001/ using document root <none> and password file <none>
I20260924 16:15:45.200040 20098 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:45.200105 20098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:45.200346 20098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:45.202566 20098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "5635eb07918149bd80ade756d0143d44"
format_stamp: "Formatted at 2026-09-24 16:15:45 on dist-test-slave-6wvm"
server_key: "6aa14cabbe593e9e5380666099d25d79"
server_key_iv: "80a5d54ffd7e90886f76a9e9df425e50"
server_key_version: "encryptionkey@0"
I20260924 16:15:45.203150 20098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "5635eb07918149bd80ade756d0143d44"
format_stamp: "Formatted at 2026-09-24 16:15:45 on dist-test-slave-6wvm"
server_key: "6aa14cabbe593e9e5380666099d25d79"
server_key_iv: "80a5d54ffd7e90886f76a9e9df425e50"
server_key_version: "encryptionkey@0"
I20260924 16:15:45.206873 20098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 16:15:45.209741 20112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:45.211637 20098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:45.211777 20098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "5635eb07918149bd80ade756d0143d44"
format_stamp: "Formatted at 2026-09-24 16:15:45 on dist-test-slave-6wvm"
server_key: "6aa14cabbe593e9e5380666099d25d79"
server_key_iv: "80a5d54ffd7e90886f76a9e9df425e50"
server_key_version: "encryptionkey@0"
I20260924 16:15:45.211894 20098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:45.240156 20098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:45.241044 20098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:45.241283 20098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:45.242004 20098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:45.243126 20098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:45.243177 20098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:45.243260 20098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:45.243307 20098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:45.252308 20098 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:36019
I20260924 16:15:45.252326 20228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:36019 every 8 connection(s)
I20260924 16:15:45.253350 20098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:45.256564 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 20098
I20260924 16:15:45.256695 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:15:45.257025 2037 external_mini_cluster.cc:1428] Setting key 408b6681947314b479aa4c4ab3f87753
I20260924 16:15:45.263622 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.255630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:37473 (local address 127.1.253.126:41731)
0924 16:15:45.255802 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:45.255806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:45.257133 (+ 1327us) server_negotiation.cc:407] Connection header received
0924 16:15:45.258821 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:45.258824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:45.258878 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:45.258967 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:45.260011 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.260539 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.261769 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.261979 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.262054 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.263187 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:45.263190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:45.263192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:45.263205 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:45.263225 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:45.263233 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:45.263291 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:45.263440 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 16:15:45.263489 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:15:45.264572 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.256034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:41731 (local address 127.1.253.67:37473)
0924 16:15:45.256904 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:45.256966 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:45.258611 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:45.259143 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:45.259151 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:45.259275 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:45.259847 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.259862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.260676 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.260679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.261659 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.261668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.262119 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.262123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.262341 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.262917 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:15:45.262919 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:45.262961 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:15:45.263028 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:45.263311 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:45.263318 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:45.263322 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:15:45.263420 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 16:15:45.263675 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":130,"threads_started":1}
I20260924 16:15:45.265739 20229 heartbeater.cc:347] Connected to a master server at 127.1.253.126:41731
I20260924 16:15:45.266047 20229 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:45.266698 20229 heartbeater.cc:511] Master 127.1.253.126:41731 requested a full tablet report, sending...
I20260924 16:15:45.267802 19770 ts_manager.cc:194] Registered new tserver with Master: 5635eb07918149bd80ade756d0143d44 (127.1.253.67:36019)
I20260924 16:15:45.268388 19770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.253.67:37473
I20260924 16:15:45.271309 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:15:45.276993 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.273289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59780 (local address 127.1.253.126:41731)
0924 16:15:45.273432 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:45.273436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:45.273593 (+ 157us) server_negotiation.cc:407] Connection header received
0924 16:15:45.273696 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:45.273698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:45.273744 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:45.273832 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:45.274102 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.274692 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.275535 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.275776 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.275867 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.276607 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:45.276611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:45.276613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:45.276625 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:45.276644 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:45.276653 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:45.276716 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:45.276791 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 16:15:45.276841 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:15:45.279901 19770 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:15:45.283011 19770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:15:45.298171 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.292940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.1.253.65:33491)
0924 16:15:45.293314 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:45.293327 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:45.293354 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:15:45.293437 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:45.293443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:45.293657 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:45.293827 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:45.294213 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.295600 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.296942 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.297487 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.297623 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.297823 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:45.297909 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 16:15:45.298004 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":136,"threads_started":1}
I20260924 16:15:45.298170 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.291805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:33491 (local address 127.0.0.1:54054)
0924 16:15:45.292539 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:45.292606 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:45.292743 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:45.293790 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:45.293796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:45.293832 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:45.294060 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.294068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.295761 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.295764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.296821 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.296828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.297632 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.297635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.297737 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.297774 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:15:45.297881 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:15:45.298000 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":73,"spinlock_wait_cycles":10112,"thread_start_us":105,"threads_started":1}
I20260924 16:15:45.298822 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.291798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.1.253.67:36019)
0924 16:15:45.293057 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:45.293072 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:45.293100 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:15:45.293182 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:45.293187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:45.293387 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:45.293560 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:45.294336 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.295159 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.296733 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.297608 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.297717 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.297908 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:45.298536 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 16:15:45.298646 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1142,"thread_start_us":151,"threads_started":1}
I20260924 16:15:45.299029 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.292364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:43881 (local address 127.0.0.1:50304)
0924 16:15:45.292633 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:45.292648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:45.292763 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:45.294327 (+ 1564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:45.294330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:45.294351 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:45.294447 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.294453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.295945 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.295952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.297258 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.297269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.298602 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.298606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.298712 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.298744 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:15:45.298799 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 16:15:45.298881 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"spinlock_wait_cycles":64768,"thread_start_us":91,"threads_started":1}
I20260924 16:15:45.299031 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.291585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:36019 (local address 127.0.0.1:38194)
0924 16:15:45.292540 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:45.292605 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:45.292747 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:45.293790 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:45.293796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:45.293831 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:45.294060 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.294068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.295287 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.295293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.296590 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.296601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.297976 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.297979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.298078 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.298105 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:15:45.298184 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 16:15:45.298290 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":40,"spinlock_wait_cycles":12928,"thread_start_us":89,"threads_started":1}
I20260924 16:15:45.299710 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.293004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50304 (local address 127.1.253.66:43881)
0924 16:15:45.293748 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:45.293764 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:45.293790 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:15:45.293866 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:45.293870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:45.294057 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:45.294314 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:45.294583 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.295776 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.297439 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.298457 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.299115 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.299316 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:45.299402 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 16:15:45.299503 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":146,"threads_started":1}
I20260924 16:15:45.301730 19894 tablet_service.cc:1467] Processing CreateTablet for tablet f4a91972b5814995941eb07b593bd64b (DEFAULT_TABLE table=test-table [id=5fac27dec47348a5b671f72528e542d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:45.302520 20028 tablet_service.cc:1467] Processing CreateTablet for tablet f4a91972b5814995941eb07b593bd64b (DEFAULT_TABLE table=test-table [id=5fac27dec47348a5b671f72528e542d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:45.302901 19894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a91972b5814995941eb07b593bd64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:45.302763 20160 tablet_service.cc:1467] Processing CreateTablet for tablet f4a91972b5814995941eb07b593bd64b (DEFAULT_TABLE table=test-table [id=5fac27dec47348a5b671f72528e542d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:45.303411 20028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a91972b5814995941eb07b593bd64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:45.303686 20160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a91972b5814995941eb07b593bd64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:45.309480 20246 tablet_bootstrap.cc:492] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2: Bootstrap starting.
I20260924 16:15:45.310889 20247 tablet_bootstrap.cc:492] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a: Bootstrap starting.
I20260924 16:15:45.311760 20248 tablet_bootstrap.cc:492] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44: Bootstrap starting.
I20260924 16:15:45.312342 20246 tablet_bootstrap.cc:654] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:45.313208 20247 tablet_bootstrap.cc:654] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:45.313299 20246 log.cc:824] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:45.313663 20248 tablet_bootstrap.cc:654] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:45.314034 20247 log.cc:824] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:45.314529 20248 log.cc:824] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:45.315699 20246 tablet_bootstrap.cc:492] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2: No bootstrap required, opened a new log
I20260924 16:15:45.315922 20246 ts_tablet_manager.cc:1397] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 16:15:45.316100 20247 tablet_bootstrap.cc:492] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a: No bootstrap required, opened a new log
I20260924 16:15:45.316318 20247 ts_tablet_manager.cc:1397] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:15:45.316488 20248 tablet_bootstrap.cc:492] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44: No bootstrap required, opened a new log
I20260924 16:15:45.316704 20248 ts_tablet_manager.cc:1397] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:15:45.320497 20248 raft_consensus.cc:348] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5635eb07918149bd80ade756d0143d44" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36019 } } peers { permanent_uuid: "58494fdc54074f879557c1cc4c97f30a" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33491 } } peers { permanent_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43881 } }
I20260924 16:15:45.320768 20248 raft_consensus.cc:374] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:45.320823 20248 raft_consensus.cc:729] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5635eb07918149bd80ade756d0143d44, State: Initialized, Role: FOLLOWER
I20260924 16:15:45.320887 20246 raft_consensus.cc:348] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5635eb07918149bd80ade756d0143d44" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36019 } } peers { permanent_uuid: "58494fdc54074f879557c1cc4c97f30a" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33491 } } peers { permanent_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43881 } }
I20260924 16:15:45.320887 20247 raft_consensus.cc:348] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5635eb07918149bd80ade756d0143d44" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36019 } } peers { permanent_uuid: "58494fdc54074f879557c1cc4c97f30a" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33491 } } peers { permanent_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43881 } }
I20260924 16:15:45.321161 20247 raft_consensus.cc:374] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:45.321161 20246 raft_consensus.cc:374] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:45.321226 20247 raft_consensus.cc:729] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58494fdc54074f879557c1cc4c97f30a, State: Initialized, Role: FOLLOWER
I20260924 16:15:45.321226 20246 raft_consensus.cc:729] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dfb921458e2468b8b47d5e33fcf4dc2, State: Initialized, Role: FOLLOWER
I20260924 16:15:45.321309 20248 consensus_queue.cc:260] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5635eb07918149bd80ade756d0143d44" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36019 } } peers { permanent_uuid: "58494fdc54074f879557c1cc4c97f30a" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33491 } } peers { permanent_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43881 } }
I20260924 16:15:45.321741 20246 consensus_queue.cc:260] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5635eb07918149bd80ade756d0143d44" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36019 } } peers { permanent_uuid: "58494fdc54074f879557c1cc4c97f30a" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33491 } } peers { permanent_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43881 } }
I20260924 16:15:45.321741 20247 consensus_queue.cc:260] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5635eb07918149bd80ade756d0143d44" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36019 } } peers { permanent_uuid: "58494fdc54074f879557c1cc4c97f30a" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33491 } } peers { permanent_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43881 } }
I20260924 16:15:45.322055 20229 heartbeater.cc:503] Master 127.1.253.126:41731 was elected leader, sending a full tablet report...
I20260924 16:15:45.322453 20248 ts_tablet_manager.cc:1428] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:15:45.322503 19961 heartbeater.cc:503] Master 127.1.253.126:41731 was elected leader, sending a full tablet report...
I20260924 16:15:45.322801 20247 ts_tablet_manager.cc:1428] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 16:15:45.323513 20095 heartbeater.cc:503] Master 127.1.253.126:41731 was elected leader, sending a full tablet report...
I20260924 16:15:45.323841 20246 ts_tablet_manager.cc:1428] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260924 16:15:45.366251 19962 tablet.cc:2369] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:15:45.451426 20253 raft_consensus.cc:482] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:45.451759 20253 raft_consensus.cc:504] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5635eb07918149bd80ade756d0143d44" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36019 } } peers { permanent_uuid: "58494fdc54074f879557c1cc4c97f30a" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33491 } } peers { permanent_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43881 } }
I20260924 16:15:45.453375 20253 leader_election.cc:288] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5635eb07918149bd80ade756d0143d44 (127.1.253.67:36019), 3dfb921458e2468b8b47d5e33fcf4dc2 (127.1.253.66:43881)
I20260924 16:15:45.458158 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.453752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:39291 (local address 127.1.253.67:36019)
0924 16:15:45.453984 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:45.453993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:45.454116 (+ 123us) server_negotiation.cc:407] Connection header received
0924 16:15:45.454409 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:45.454415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:45.454475 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:45.454554 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:45.455032 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.455662 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.457067 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.457463 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.457559 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.457646 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:45.457963 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 16:15:45.458038 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 16:15:45.458173 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.454175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:43881 (local address 127.1.253.65:43293)
0924 16:15:45.454606 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:45.454621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:45.454743 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:45.455029 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:45.455032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:45.455050 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:45.455180 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.455187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.456079 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.456082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.456884 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.456893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.457699 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.457702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.457786 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.457803 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 16:15:45.457865 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 16:15:45.457963 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":28,"spinlock_wait_cycles":8064,"thread_start_us":167,"threads_started":1}
I20260924 16:15:45.458163 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.454388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:43293 (local address 127.1.253.66:43881)
0924 16:15:45.454580 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:45.454586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:45.454721 (+ 135us) server_negotiation.cc:407] Connection header received
0924 16:15:45.454836 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:45.454840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:45.454903 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:45.455003 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:45.455311 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.455951 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.457040 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.457490 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.457668 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.457736 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:45.457929 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 16:15:45.458012 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 16:15:45.458205 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.453631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:36019 (local address 127.1.253.65:39291)
0924 16:15:45.453996 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:45.454010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:45.454182 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:45.454610 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:45.454615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:45.454644 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:45.454864 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.454874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.455804 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.455808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.456883 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:45.456894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.457617 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.457622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:45.457750 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.457790 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:15:45.457930 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 16:15:45.458012 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":7,"thread_start_us":93,"threads_started":1}
I20260924 16:15:45.458896 20181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a91972b5814995941eb07b593bd64b" candidate_uuid: "58494fdc54074f879557c1cc4c97f30a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5635eb07918149bd80ade756d0143d44" is_pre_election: true
I20260924 16:15:45.458899 20048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a91972b5814995941eb07b593bd64b" candidate_uuid: "58494fdc54074f879557c1cc4c97f30a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2" is_pre_election: true
I20260924 16:15:45.459241 20181 raft_consensus.cc:2454] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58494fdc54074f879557c1cc4c97f30a in term 0.
I20260924 16:15:45.459249 20048 raft_consensus.cc:2454] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58494fdc54074f879557c1cc4c97f30a in term 0.
I20260924 16:15:45.459810 19849 leader_election.cc:302] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dfb921458e2468b8b47d5e33fcf4dc2, 58494fdc54074f879557c1cc4c97f30a; no voters:
I20260924 16:15:45.460075 20253 raft_consensus.cc:2781] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:15:45.460165 20253 raft_consensus.cc:482] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:15:45.460209 20253 raft_consensus.cc:3037] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:45.461274 20253 raft_consensus.cc:504] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5635eb07918149bd80ade756d0143d44" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36019 } } peers { permanent_uuid: "58494fdc54074f879557c1cc4c97f30a" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33491 } } peers { permanent_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43881 } }
I20260924 16:15:45.461634 20253 leader_election.cc:288] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [CANDIDATE]: Term 1 election: Requested vote from peers 5635eb07918149bd80ade756d0143d44 (127.1.253.67:36019), 3dfb921458e2468b8b47d5e33fcf4dc2 (127.1.253.66:43881)
I20260924 16:15:45.462067 20048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a91972b5814995941eb07b593bd64b" candidate_uuid: "58494fdc54074f879557c1cc4c97f30a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2"
I20260924 16:15:45.462113 20181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a91972b5814995941eb07b593bd64b" candidate_uuid: "58494fdc54074f879557c1cc4c97f30a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5635eb07918149bd80ade756d0143d44"
I20260924 16:15:45.462275 20181 raft_consensus.cc:3037] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:45.462230 20048 raft_consensus.cc:3037] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:45.463322 20181 raft_consensus.cc:2454] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58494fdc54074f879557c1cc4c97f30a in term 1.
I20260924 16:15:45.463539 20048 raft_consensus.cc:2454] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58494fdc54074f879557c1cc4c97f30a in term 1.
I20260924 16:15:45.463624 19849 leader_election.cc:302] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5635eb07918149bd80ade756d0143d44, 58494fdc54074f879557c1cc4c97f30a; no voters:
I20260924 16:15:45.463784 20253 raft_consensus.cc:2781] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:45.464039 20253 raft_consensus.cc:686] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 1 LEADER]: Becoming Leader. State: Replica: 58494fdc54074f879557c1cc4c97f30a, State: Running, Role: LEADER
I20260924 16:15:45.464421 20253 consensus_queue.cc:237] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5635eb07918149bd80ade756d0143d44" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36019 } } peers { permanent_uuid: "58494fdc54074f879557c1cc4c97f30a" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33491 } } peers { permanent_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43881 } }
I20260924 16:15:45.467835 19767 catalog_manager.cc:5495] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a reported cstate change: term changed from 0 to 1, leader changed from <none> to 58494fdc54074f879557c1cc4c97f30a (127.1.253.65). New cstate: current_term: 1 leader_uuid: "58494fdc54074f879557c1cc4c97f30a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5635eb07918149bd80ade756d0143d44" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58494fdc54074f879557c1cc4c97f30a" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43881 } health_report { overall_health: UNKNOWN } } }
I20260924 16:15:45.492108 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.488399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.1.253.65:33491)
0924 16:15:45.488573 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:45.488580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:45.488611 (+ 31us) server_negotiation.cc:407] Connection header received
0924 16:15:45.488691 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:45.488693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:45.488752 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:45.488878 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:15:45.489149 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.489662 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.490574 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:45.490796 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:45.490916 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:45.491257 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:15:45.491710 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:15:45.491861 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:45.491899 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 16:15:45.491955 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:15:45.498492 20181 raft_consensus.cc:1261] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44 [term 1 FOLLOWER]: Refusing update from remote peer 58494fdc54074f879557c1cc4c97f30a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:45.498492 20048 raft_consensus.cc:1261] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2 [term 1 FOLLOWER]: Refusing update from remote peer 58494fdc54074f879557c1cc4c97f30a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:45.499313 20253 consensus_queue.cc:1035] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5635eb07918149bd80ade756d0143d44" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 36019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:45.499583 20257 consensus_queue.cc:1035] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dfb921458e2468b8b47d5e33fcf4dc2" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:15:45.504505 20230 tablet.cc:2369] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:15:45.509320 20261 mvcc.cc:205] Tried to move back new op lower bound from 7332931770353549312 to 7332931770225135616. Current Snapshot: MvccSnapshot[applied={T|T < 7332931770353549312}]
I20260924 16:15:45.518510 19767 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:45.518700 19767 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:45.521544 19767 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe3726a3893742f5b3132b137b5c5fc0: Sending DeleteTablet for 3 replicas of tablet f4a91972b5814995941eb07b593bd64b
I20260924 16:15:45.522413 19894 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a91972b5814995941eb07b593bd64b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:45 UTC) from {username='slave'} at 127.0.0.1:54054
I20260924 16:15:45.522518 20028 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a91972b5814995941eb07b593bd64b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:45 UTC) from {username='slave'} at 127.0.0.1:50304
I20260924 16:15:45.522962 20270 tablet_replica.cc:330] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a: stopping tablet replica
I20260924 16:15:45.522962 20271 tablet_replica.cc:330] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2: stopping tablet replica
I20260924 16:15:45.522989 20160 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a91972b5814995941eb07b593bd64b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:45 UTC) from {username='slave'} at 127.0.0.1:38194
I20260924 16:15:45.523274 20271 raft_consensus.cc:2229] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:45.523285 20270 raft_consensus.cc:2229] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:15:45.523617 20271 raft_consensus.cc:2258] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:45.523789 20270 raft_consensus.cc:2258] T f4a91972b5814995941eb07b593bd64b P 58494fdc54074f879557c1cc4c97f30a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:45.524178 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 19831
I20260924 16:15:45.525235 20271 ts_tablet_manager.cc:1905] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:45.526815 20272 tablet_replica.cc:330] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44: stopping tablet replica
I20260924 16:15:45.527040 20272 raft_consensus.cc:2229] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:45.527341 20272 raft_consensus.cc:2258] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:45.528801 20272 ts_tablet_manager.cc:1905] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:45.528854 20271 ts_tablet_manager.cc:1918] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:45.528964 20271 log.cc:1196] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/f4a91972b5814995941eb07b593bd64b
I20260924 16:15:45.529282 20271 ts_tablet_manager.cc:1939] T f4a91972b5814995941eb07b593bd64b P 3dfb921458e2468b8b47d5e33fcf4dc2: Deleting consensus metadata
I20260924 16:15:45.530651 19757 catalog_manager.cc:4841] TS 3dfb921458e2468b8b47d5e33fcf4dc2 (127.1.253.66:43881): tablet f4a91972b5814995941eb07b593bd64b (table test-table [id=5fac27dec47348a5b671f72528e542d4]) successfully deleted
I20260924 16:15:45.531327 20272 ts_tablet_manager.cc:1918] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:45.531445 20272 log.cc:1196] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/f4a91972b5814995941eb07b593bd64b
I20260924 16:15:45.531715 20272 ts_tablet_manager.cc:1939] T f4a91972b5814995941eb07b593bd64b P 5635eb07918149bd80ade756d0143d44: Deleting consensus metadata
I20260924 16:15:45.532737 19757 catalog_manager.cc:4841] TS 5635eb07918149bd80ade756d0143d44 (127.1.253.67:36019): tablet f4a91972b5814995941eb07b593bd64b (table test-table [id=5fac27dec47348a5b671f72528e542d4]) successfully deleted
W20260924 16:15:45.533736 19756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:33491 (error 108)
I20260924 16:15:45.533798 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 19964
I20260924 16:15:45.535108 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.534840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:33491 (local address 127.0.0.1:54072)
0924 16:15:45.534978 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:45.535040 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 16:15:45.535274 19756 catalog_manager.cc:4584] TS 58494fdc54074f879557c1cc4c97f30a (127.1.253.65:33491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4a91972b5814995941eb07b593bd64b: Network error: Client connection negotiation failed: client connection to 127.1.253.65:33491: connect: Connection refused (error 111)
I20260924 16:15:45.541164 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 20098
I20260924 16:15:45.547338 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 19739
2026-09-24T16:15:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1182 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:45 dist-test-slave-6wvm krb5kdc[20276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:45 dist-test-slave-6wvm krb5kdc[20276](info): set up 2 sockets
Sep 24 16:15:45 dist-test-slave-6wvm krb5kdc[20276](info): commencing operation
krb5kdc: starting...
W20260924 16:15:46.641475 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:15:46 dist-test-slave-6wvm krb5kdc[20276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:15:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:15:46.773620 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:42321
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:42321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:46.887351 20289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:46.887641 20289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:46.887682 20289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:46.891888 20289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:46.891956 20289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:46.891976 20289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:46.891993 20289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:46.892009 20289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:46.897423 20289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:42321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:42321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:46.898605 20289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:46.899688 20289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:46.908360 20295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:46.908398 20294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:46.908593 20289 server_base.cc:1034] running on GCE node
W20260924 16:15:46.908363 20297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:46.909174 20289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:46.910326 20289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:46.911520 20289 hybrid_clock.cc:648] HybridClock initialized: now 1790266546911509 us; error 30 us; skew 500 ppm
Sep 24 16:15:46 dist-test-slave-6wvm krb5kdc[20276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266546, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:46.914206 20289 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:15:46.915161 20289 webserver.cc:460] Webserver started at http://127.1.253.126:44607/ using document root <none> and password file <none>
I20260924 16:15:46.915736 20289 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:46.915834 20289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:46.916054 20289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:46.918128 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "041a36d972894754bdc57a6098309d09"
format_stamp: "Formatted at 2026-09-24 16:15:46 on dist-test-slave-6wvm"
server_key: "36a7e3b5b201e220e3a526d0ddfca0dd"
server_key_iv: "3779dfb691034e666a74aaeafd239b93"
server_key_version: "encryptionkey@0"
I20260924 16:15:46.918781 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "041a36d972894754bdc57a6098309d09"
format_stamp: "Formatted at 2026-09-24 16:15:46 on dist-test-slave-6wvm"
server_key: "36a7e3b5b201e220e3a526d0ddfca0dd"
server_key_iv: "3779dfb691034e666a74aaeafd239b93"
server_key_version: "encryptionkey@0"
I20260924 16:15:46.922787 20289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:15:46.925191 20304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:46.926533 20289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:15:46.926678 20289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "041a36d972894754bdc57a6098309d09"
format_stamp: "Formatted at 2026-09-24 16:15:46 on dist-test-slave-6wvm"
server_key: "36a7e3b5b201e220e3a526d0ddfca0dd"
server_key_iv: "3779dfb691034e666a74aaeafd239b93"
server_key_version: "encryptionkey@0"
I20260924 16:15:46.926795 20289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:46.952057 20289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:46.955149 20289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:46.955349 20289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:46.963034 20289 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:42321
I20260924 16:15:46.963059 20356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:42321 every 8 connection(s)
I20260924 16:15:46.964030 20289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:46.966777 20359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:46.970490 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 20289
I20260924 16:15:46.970641 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:15:46.970959 2037 external_mini_cluster.cc:1428] Setting key 1c8dc99f982bc80ac98f0cfaf7d68af7
I20260924 16:15:46.973026 20359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09: Bootstrap starting.
I20260924 16:15:46.975263 20359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:46.976231 20359 log.cc:824] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:46.978188 20359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09: No bootstrap required, opened a new log
Sep 24 16:15:46 dist-test-slave-6wvm krb5kdc[20276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:15:46.982321 20359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041a36d972894754bdc57a6098309d09" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42321 } }
I20260924 16:15:46.982544 20359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:46.982625 20359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041a36d972894754bdc57a6098309d09, State: Initialized, Role: FOLLOWER
I20260924 16:15:46.983114 20359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041a36d972894754bdc57a6098309d09" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42321 } }
I20260924 16:15:46.983294 20359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:46.983372 20359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:46.983517 20359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:46.984405 20359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041a36d972894754bdc57a6098309d09" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42321 } }
I20260924 16:15:46.984760 20359 leader_election.cc:302] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 041a36d972894754bdc57a6098309d09; no voters:
I20260924 16:15:46.985127 20359 leader_election.cc:288] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:46.985236 20364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:46.985416 20364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [term 1 LEADER]: Becoming Leader. State: Replica: 041a36d972894754bdc57a6098309d09, State: Running, Role: LEADER
I20260924 16:15:46.985716 20364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041a36d972894754bdc57a6098309d09" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42321 } }
I20260924 16:15:46.986498 20359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:46.987023 20366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 041a36d972894754bdc57a6098309d09. Latest consensus state: current_term: 1 leader_uuid: "041a36d972894754bdc57a6098309d09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041a36d972894754bdc57a6098309d09" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42321 } } }
I20260924 16:15:46.987162 20366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:46.987411 20365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "041a36d972894754bdc57a6098309d09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041a36d972894754bdc57a6098309d09" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42321 } } }
I20260924 16:15:46.987514 20365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:46.988011 20369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:46.988189 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:46.972462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.1.253.126:42321)
0924 16:15:46.972983 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:46.973031 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:46.973076 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:15:46.974165 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:46.974197 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:46.974837 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:46.975265 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:46.976513 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:46.977708 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:46.978766 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:46.979255 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:46.979445 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:46.982223 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:46.982229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:46.982295 (+ 66us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:46.982331 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:46.984209 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:46.984721 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:46.984727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:46.984733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:46.984889 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:46.985273 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:46.985277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:46.985279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:46.985870 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:46.986103 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:46.986867 (+ 764us) server_negotiation.cc:299] Negotiation successful
0924 16:15:46.987192 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":130,"threads_started":1}
I20260924 16:15:46.991712 20369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:46.997714 20369 catalog_manager.cc:1347] Generated new cluster ID: a52525dc5f1d407797e463adb5eeaed1
I20260924 16:15:46.997794 20369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:47.007328 20369 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:47.008244 20369 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:47.017877 20369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09: Generated new TSK 0
I20260924 16:15:47.018646 20369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:15:47.049187 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:42321
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:47.163636 20385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:47.163892 20385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:47.163935 20385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:47.168042 20385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:47.168109 20385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:47.168165 20385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:15:47.173204 20385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:42321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:47.174283 20385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:47.175434 20385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:47.185328 20390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:47.185385 20385 server_base.cc:1034] running on GCE node
W20260924 16:15:47.185295 20393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:47.185350 20391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:47.185978 20385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:47.186730 20385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:47.187924 20385 hybrid_clock.cc:648] HybridClock initialized: now 1790266547187903 us; error 39 us; skew 500 ppm
Sep 24 16:15:47 dist-test-slave-6wvm krb5kdc[20276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266547, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:47.190908 20385 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:15:47.191720 20385 webserver.cc:460] Webserver started at http://127.1.253.65:36341/ using document root <none> and password file <none>
I20260924 16:15:47.192261 20385 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:47.192318 20385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:47.192477 20385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:47.194530 20385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "88bf283162df4b86943b3958377a8880"
format_stamp: "Formatted at 2026-09-24 16:15:47 on dist-test-slave-6wvm"
server_key: "e2ff812fb2cbcef6ee4c9df88d026455"
server_key_iv: "d32739dc179578df0fe5de5ade04b78c"
server_key_version: "encryptionkey@0"
I20260924 16:15:47.195088 20385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "88bf283162df4b86943b3958377a8880"
format_stamp: "Formatted at 2026-09-24 16:15:47 on dist-test-slave-6wvm"
server_key: "e2ff812fb2cbcef6ee4c9df88d026455"
server_key_iv: "d32739dc179578df0fe5de5ade04b78c"
server_key_version: "encryptionkey@0"
I20260924 16:15:47.198904 20385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:15:47.201514 20401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:47.202960 20385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:47.203092 20385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "88bf283162df4b86943b3958377a8880"
format_stamp: "Formatted at 2026-09-24 16:15:47 on dist-test-slave-6wvm"
server_key: "e2ff812fb2cbcef6ee4c9df88d026455"
server_key_iv: "d32739dc179578df0fe5de5ade04b78c"
server_key_version: "encryptionkey@0"
I20260924 16:15:47.203200 20385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:47.250137 20385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:47.253997 20385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:47.254220 20385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:47.254899 20385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:47.255779 20385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:47.255826 20385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:47.255903 20385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:47.255944 20385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:47.265018 20385 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:40447
I20260924 16:15:47.265039 20518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:40447 every 8 connection(s)
I20260924 16:15:47.265976 20385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:47.266577 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 20385
I20260924 16:15:47.266734 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:15:47.267072 2037 external_mini_cluster.cc:1428] Setting key c8d5ab0598e1e4dcc466b7d2a7284e7f
Sep 24 16:15:47 dist-test-slave-6wvm krb5kdc[20276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266547, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
I20260924 16:15:47.281725 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.268168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:33163 (local address 127.1.253.126:42321)
0924 16:15:47.268315 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:47.268320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:47.269450 (+ 1130us) server_negotiation.cc:407] Connection header received
0924 16:15:47.270710 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:47.270715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:47.270784 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:47.270868 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:47.272527 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.273408 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.274637 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.275076 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.275207 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.277918 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:47.277921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:47.277923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:47.277948 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:47.279309 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:47.279918 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:47.279922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:47.279924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:47.280010 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:47.280388 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:47.280392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:47.280397 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:47.280699 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:47.280811 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:47.281474 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 16:15:47.281586 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:15:47.282850 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.268665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42321 (local address 127.1.253.65:33163)
0924 16:15:47.269242 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:47.269316 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:47.270415 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:47.271107 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:47.271117 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:47.271647 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:47.272364 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.272381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.273544 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.273546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.274510 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.274519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.275236 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.275239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.275386 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.276018 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:47.276035 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:47.277715 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:47.279436 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:47.279442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:47.279456 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:47.279789 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:47.280117 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:47.280120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:47.280122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:47.280268 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:47.280847 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:47.280856 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:47.281195 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 16:15:47.281444 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 16:15:47.281821 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":139,"threads_started":1}
I20260924 16:15:47.284191 20519 heartbeater.cc:347] Connected to a master server at 127.1.253.126:42321
I20260924 16:15:47.284456 20519 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:47.285036 20519 heartbeater.cc:511] Master 127.1.253.126:42321 requested a full tablet report, sending...
I20260924 16:15:47.286692 20321 ts_manager.cc:194] Registered new tserver with Master: 88bf283162df4b86943b3958377a8880 (127.1.253.65:40447)
I20260924 16:15:47.287911 20321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:33163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:15:47.297981 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:42321
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:47.412760 20525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:47.413056 20525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:47.413102 20525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:47.417306 20525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:47.417375 20525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:47.417438 20525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:15:47.423280 20525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:42321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:47.424480 20525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:47.425628 20525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:47.435366 20531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:47.435396 20530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:47.435554 20525 server_base.cc:1034] running on GCE node
W20260924 16:15:47.435392 20533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:47.435961 20525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:47.436586 20525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:47.437783 20525 hybrid_clock.cc:648] HybridClock initialized: now 1790266547437770 us; error 27 us; skew 500 ppm
Sep 24 16:15:47 dist-test-slave-6wvm krb5kdc[20276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266547, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:47.440596 20525 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:15:47.441418 20525 webserver.cc:460] Webserver started at http://127.1.253.66:45137/ using document root <none> and password file <none>
I20260924 16:15:47.441941 20525 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:47.441998 20525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:47.442157 20525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:47.444269 20525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "d321970c59d0449c905ccf4ef0a798c3"
format_stamp: "Formatted at 2026-09-24 16:15:47 on dist-test-slave-6wvm"
server_key: "1c5e4e915edc18abb62e8792c1cd1e85"
server_key_iv: "548cf9dc9b1c70f0464f6c537c646037"
server_key_version: "encryptionkey@0"
I20260924 16:15:47.444787 20525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "d321970c59d0449c905ccf4ef0a798c3"
format_stamp: "Formatted at 2026-09-24 16:15:47 on dist-test-slave-6wvm"
server_key: "1c5e4e915edc18abb62e8792c1cd1e85"
server_key_iv: "548cf9dc9b1c70f0464f6c537c646037"
server_key_version: "encryptionkey@0"
I20260924 16:15:47.448355 20525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:15:47.450740 20540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:47.451911 20525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:15:47.452047 20525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "d321970c59d0449c905ccf4ef0a798c3"
format_stamp: "Formatted at 2026-09-24 16:15:47 on dist-test-slave-6wvm"
server_key: "1c5e4e915edc18abb62e8792c1cd1e85"
server_key_iv: "548cf9dc9b1c70f0464f6c537c646037"
server_key_version: "encryptionkey@0"
I20260924 16:15:47.452157 20525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:47.476468 20525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:47.479588 20525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:47.479799 20525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:47.480391 20525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:47.481271 20525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:47.481359 20525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:47.481442 20525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:47.481498 20525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:47.491050 20525 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:42187
I20260924 16:15:47.491122 20657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:42187 every 8 connection(s)
I20260924 16:15:47.492000 20525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:47.495045 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 20525
I20260924 16:15:47.495199 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:15:47.495497 2037 external_mini_cluster.cc:1428] Setting key 367464bb74f632819c04adb8ebe734af
Sep 24 16:15:47 dist-test-slave-6wvm krb5kdc[20276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266547, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:15:47.505962 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.493902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:58205 (local address 127.1.253.126:42321)
0924 16:15:47.494047 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:47.494050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:47.494953 (+ 903us) server_negotiation.cc:407] Connection header received
0924 16:15:47.496038 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:47.496042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:47.496101 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:47.496184 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:47.497357 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.497896 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.499169 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.499396 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.499518 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.502283 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:47.502289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:47.502296 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:47.502320 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:47.503768 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:47.504306 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:47.504310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:47.504312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:47.504395 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:47.504760 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:47.504764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:47.504766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:47.504958 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:47.505075 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:47.505660 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 16:15:47.505791 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:15:47.506927 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.494347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42321 (local address 127.1.253.66:58205)
0924 16:15:47.494798 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:47.494841 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:47.495818 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:47.496361 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:47.496369 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:47.496717 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:47.497205 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.497219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.498034 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.498038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.498996 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.499006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.499559 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.499563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.499746 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.500374 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:47.500390 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:47.502077 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:47.503898 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:47.503904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:47.503919 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:47.504192 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:47.504498 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:47.504502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:47.504504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:47.504653 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:47.505078 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:47.505087 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:47.505403 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 16:15:47.505610 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 16:15:47.505942 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
I20260924 16:15:47.508314 20658 heartbeater.cc:347] Connected to a master server at 127.1.253.126:42321
I20260924 16:15:47.508623 20658 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:47.509299 20658 heartbeater.cc:511] Master 127.1.253.126:42321 requested a full tablet report, sending...
I20260924 16:15:47.510610 20321 ts_manager.cc:194] Registered new tserver with Master: d321970c59d0449c905ccf4ef0a798c3 (127.1.253.66:42187)
I20260924 16:15:47.511229 20321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:58205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:15:47.525230 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:42321
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:47.642459 20663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:47.642765 20663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:47.642848 20663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:47.646895 20663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:47.647001 20663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:47.647099 20663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:15:47.652499 20663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:42321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:47.653651 20663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:47.654811 20663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:47.665391 20669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:47.665373 20671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:47.665373 20668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:47.665776 20663 server_base.cc:1034] running on GCE node
I20260924 16:15:47.666230 20663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:47.666952 20663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:47.668152 20663 hybrid_clock.cc:648] HybridClock initialized: now 1790266547668135 us; error 31 us; skew 500 ppm
Sep 24 16:15:47 dist-test-slave-6wvm krb5kdc[20276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266547, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:47.670892 20663 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:15:47.671783 20663 webserver.cc:460] Webserver started at http://127.1.253.67:38527/ using document root <none> and password file <none>
I20260924 16:15:47.672377 20663 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:47.672487 20663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:47.672735 20663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:47.674899 20663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "ef693e68c7b4405ab119641e361347a3"
format_stamp: "Formatted at 2026-09-24 16:15:47 on dist-test-slave-6wvm"
server_key: "87b85e5cdc3357d52984be2c6fae0c05"
server_key_iv: "cbf7f98e309545fc6d3bf91465a31bf2"
server_key_version: "encryptionkey@0"
I20260924 16:15:47.675486 20663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "ef693e68c7b4405ab119641e361347a3"
format_stamp: "Formatted at 2026-09-24 16:15:47 on dist-test-slave-6wvm"
server_key: "87b85e5cdc3357d52984be2c6fae0c05"
server_key_iv: "cbf7f98e309545fc6d3bf91465a31bf2"
server_key_version: "encryptionkey@0"
I20260924 16:15:47.679345 20663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:15:47.682000 20678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:47.683418 20663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:15:47.683588 20663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "ef693e68c7b4405ab119641e361347a3"
format_stamp: "Formatted at 2026-09-24 16:15:47 on dist-test-slave-6wvm"
server_key: "87b85e5cdc3357d52984be2c6fae0c05"
server_key_iv: "cbf7f98e309545fc6d3bf91465a31bf2"
server_key_version: "encryptionkey@0"
I20260924 16:15:47.683715 20663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:47.704739 20663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:47.708577 20663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:47.708750 20663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:47.709328 20663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:47.710188 20663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:47.710302 20663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:47.710366 20663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:47.710388 20663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:47.720496 20663 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:44011
I20260924 16:15:47.720556 20794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:44011 every 8 connection(s)
I20260924 16:15:47.721571 20663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:47.722563 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 20663
I20260924 16:15:47.722715 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:15:47.723021 2037 external_mini_cluster.cc:1428] Setting key ad927476f6197dff03ae94064584262f
Sep 24 16:15:47 dist-test-slave-6wvm krb5kdc[20276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266547, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:15:47.736120 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.723859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:40901 (local address 127.1.253.126:42321)
0924 16:15:47.724061 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:47.724069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:47.725346 (+ 1277us) server_negotiation.cc:407] Connection header received
0924 16:15:47.726398 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:47.726401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:47.726455 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:47.726531 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:47.727709 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.728244 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.729430 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.729673 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.729774 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.732526 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:47.732530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:47.732532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:47.732558 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:47.733935 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:47.734551 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:47.734554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:47.734556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:47.734642 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:47.735024 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:47.735027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:47.735028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:47.735200 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:47.735276 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:47.735858 (+ 582us) server_negotiation.cc:299] Negotiation successful
0924 16:15:47.735991 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:15:47.737257 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.724421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42321 (local address 127.1.253.67:40901)
0924 16:15:47.725179 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:47.725223 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:47.726128 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:47.726675 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:47.726682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:47.727050 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:47.727556 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.727571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.728363 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.728365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.729264 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.729274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.729828 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.729830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.729986 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.730723 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:47.730740 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:47.732353 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:47.734065 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:47.734070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:47.734083 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:47.734454 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:47.734740 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:47.734742 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:47.734743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:47.734928 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:47.735325 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:47.735330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:47.735573 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 16:15:47.735770 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 16:15:47.736164 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":169,"threads_started":1}
I20260924 16:15:47.738504 20796 heartbeater.cc:347] Connected to a master server at 127.1.253.126:42321
I20260924 16:15:47.738775 20796 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:47.739282 20796 heartbeater.cc:511] Master 127.1.253.126:42321 requested a full tablet report, sending...
I20260924 16:15:47.740324 20321 ts_manager.cc:194] Registered new tserver with Master: ef693e68c7b4405ab119641e361347a3 (127.1.253.67:44011)
I20260924 16:15:47.740862 20321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:40901
I20260924 16:15:47.749455 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:15:47.759348 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.751634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.1.253.126:42321)
0924 16:15:47.751786 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:47.751790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:47.751953 (+ 163us) server_negotiation.cc:407] Connection header received
0924 16:15:47.752017 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:47.752020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:47.752061 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:47.752126 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:47.752790 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.753301 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.754131 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.754402 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.754486 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.756115 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:47.756119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:47.756120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:47.756143 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:47.757601 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:47.758069 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:47.758072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:47.758073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:47.758147 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:47.758565 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:47.758568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:47.758569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:47.758775 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:47.758896 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:47.759153 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 16:15:47.759231 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:15:47.762727 20321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:15:47.765950 20321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:15:47.780711 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.774825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.1.253.66:42187)
0924 16:15:47.775242 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:47.775256 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:47.775282 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:15:47.775468 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:47.775472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:47.775706 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:47.775878 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:47.776489 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.777599 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.778833 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.779679 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.779798 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.779951 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:47.780433 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 16:15:47.780528 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":137,"threads_started":1}
I20260924 16:15:47.780697 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.774607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:42187 (local address 127.0.0.1:47538)
0924 16:15:47.775042 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:47.775114 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:47.775260 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:47.776138 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:47.776144 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:47.776170 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:47.776326 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.776334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.777742 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.777747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.778705 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.778715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.780009 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.780012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.780230 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.780287 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 16:15:47.780400 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 16:15:47.780523 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":82,"thread_start_us":99,"threads_started":1}
I20260924 16:15:47.781685 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.775543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:44011 (local address 127.0.0.1:47454)
0924 16:15:47.775818 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:47.775833 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:47.775920 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:47.777062 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:47.777065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:47.777082 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:47.777184 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.777191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.778875 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.778878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.780218 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.780229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.781291 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.781295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.781393 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.781416 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:15:47.781490 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 16:15:47.781558 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":111,"thread_start_us":136,"threads_started":1}
I20260924 16:15:47.781798 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.775723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.1.253.67:44011)
0924 16:15:47.776555 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:47.776568 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:47.776592 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:15:47.776672 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:47.776678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:47.776897 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:47.777340 (+ 443us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:47.777506 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.778710 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.780327 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.781138 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.781283 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.781398 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:47.781536 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 16:15:47.781649 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":124,"threads_started":1}
I20260924 16:15:47.782218 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.776313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.1.253.65:40447)
0924 16:15:47.776703 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:47.776717 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:47.776741 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:15:47.777005 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:47.777015 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:47.777221 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:47.777364 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:47.777607 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.778718 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.780265 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.781138 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.781257 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.781398 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:47.782019 (+ 621us) server_negotiation.cc:299] Negotiation successful
0924 16:15:47.782099 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":95,"threads_started":1}
I20260924 16:15:47.782774 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.775917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40447 (local address 127.0.0.1:37374)
0924 16:15:47.776229 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:47.776245 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:47.776330 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:47.777380 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:47.777383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:47.777403 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:47.777481 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.777486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.778870 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.778876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.780132 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.780142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.781461 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.781465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.781556 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.781578 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:15:47.781645 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 16:15:47.781712 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":105,"thread_start_us":140,"threads_started":1}
I20260924 16:15:47.784399 20590 tablet_service.cc:1467] Processing CreateTablet for tablet b13d07a594334fc991c0c4771e33d03d (DEFAULT_TABLE table=test-table [id=fcada004f1f24dbb8c9db4274297e3d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:47.784615 20728 tablet_service.cc:1467] Processing CreateTablet for tablet b13d07a594334fc991c0c4771e33d03d (DEFAULT_TABLE table=test-table [id=fcada004f1f24dbb8c9db4274297e3d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:47.785480 20590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13d07a594334fc991c0c4771e33d03d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:47.785532 20728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13d07a594334fc991c0c4771e33d03d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:47.791868 20814 tablet_bootstrap.cc:492] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3: Bootstrap starting.
I20260924 16:15:47.792690 20813 tablet_bootstrap.cc:492] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3: Bootstrap starting.
I20260924 16:15:47.794162 20814 tablet_bootstrap.cc:654] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:47.794663 20813 tablet_bootstrap.cc:654] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:47.795507 20814 log.cc:824] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:47.795707 20813 log.cc:824] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:47.796315 20451 tablet_service.cc:1467] Processing CreateTablet for tablet b13d07a594334fc991c0c4771e33d03d (DEFAULT_TABLE table=test-table [id=fcada004f1f24dbb8c9db4274297e3d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:47.797426 20451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13d07a594334fc991c0c4771e33d03d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:47.797678 20813 tablet_bootstrap.cc:492] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3: No bootstrap required, opened a new log
I20260924 16:15:47.797678 20814 tablet_bootstrap.cc:492] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3: No bootstrap required, opened a new log
I20260924 16:15:47.797880 20813 ts_tablet_manager.cc:1397] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 16:15:47.797880 20814 ts_tablet_manager.cc:1397] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:15:47.801949 20814 raft_consensus.cc:348] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d321970c59d0449c905ccf4ef0a798c3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42187 } } peers { permanent_uuid: "ef693e68c7b4405ab119641e361347a3" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44011 } } peers { permanent_uuid: "88bf283162df4b86943b3958377a8880" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40447 } }
I20260924 16:15:47.801949 20813 raft_consensus.cc:348] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d321970c59d0449c905ccf4ef0a798c3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42187 } } peers { permanent_uuid: "ef693e68c7b4405ab119641e361347a3" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44011 } } peers { permanent_uuid: "88bf283162df4b86943b3958377a8880" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40447 } }
I20260924 16:15:47.802278 20814 raft_consensus.cc:374] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:47.802278 20813 raft_consensus.cc:374] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:47.802336 20814 raft_consensus.cc:729] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef693e68c7b4405ab119641e361347a3, State: Initialized, Role: FOLLOWER
I20260924 16:15:47.802336 20813 raft_consensus.cc:729] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d321970c59d0449c905ccf4ef0a798c3, State: Initialized, Role: FOLLOWER
I20260924 16:15:47.802838 20813 consensus_queue.cc:260] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d321970c59d0449c905ccf4ef0a798c3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42187 } } peers { permanent_uuid: "ef693e68c7b4405ab119641e361347a3" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44011 } } peers { permanent_uuid: "88bf283162df4b86943b3958377a8880" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40447 } }
I20260924 16:15:47.802838 20814 consensus_queue.cc:260] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d321970c59d0449c905ccf4ef0a798c3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42187 } } peers { permanent_uuid: "ef693e68c7b4405ab119641e361347a3" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44011 } } peers { permanent_uuid: "88bf283162df4b86943b3958377a8880" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40447 } }
I20260924 16:15:47.803586 20658 heartbeater.cc:503] Master 127.1.253.126:42321 was elected leader, sending a full tablet report...
I20260924 16:15:47.803815 20796 heartbeater.cc:503] Master 127.1.253.126:42321 was elected leader, sending a full tablet report...
I20260924 16:15:47.803895 20813 ts_tablet_manager.cc:1428] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:15:47.805531 20814 ts_tablet_manager.cc:1428] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260924 16:15:47.805774 20819 tablet_bootstrap.cc:492] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880: Bootstrap starting.
I20260924 16:15:47.808152 20819 tablet_bootstrap.cc:654] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:47.809185 20819 log.cc:824] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:47.811316 20819 tablet_bootstrap.cc:492] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880: No bootstrap required, opened a new log
I20260924 16:15:47.811551 20819 ts_tablet_manager.cc:1397] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:15:47.815269 20819 raft_consensus.cc:348] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d321970c59d0449c905ccf4ef0a798c3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42187 } } peers { permanent_uuid: "ef693e68c7b4405ab119641e361347a3" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44011 } } peers { permanent_uuid: "88bf283162df4b86943b3958377a8880" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40447 } }
I20260924 16:15:47.815611 20819 raft_consensus.cc:374] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:47.815682 20819 raft_consensus.cc:729] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88bf283162df4b86943b3958377a8880, State: Initialized, Role: FOLLOWER
I20260924 16:15:47.816162 20819 consensus_queue.cc:260] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d321970c59d0449c905ccf4ef0a798c3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42187 } } peers { permanent_uuid: "ef693e68c7b4405ab119641e361347a3" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44011 } } peers { permanent_uuid: "88bf283162df4b86943b3958377a8880" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40447 } }
I20260924 16:15:47.816912 20519 heartbeater.cc:503] Master 127.1.253.126:42321 was elected leader, sending a full tablet report...
I20260924 16:15:47.817106 20819 ts_tablet_manager.cc:1428] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:15:47.972608 20817 raft_consensus.cc:482] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 16:15:47.972860 20797 tablet.cc:2369] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:15:47.972882 20817 raft_consensus.cc:504] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d321970c59d0449c905ccf4ef0a798c3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42187 } } peers { permanent_uuid: "ef693e68c7b4405ab119641e361347a3" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44011 } } peers { permanent_uuid: "88bf283162df4b86943b3958377a8880" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40447 } }
I20260924 16:15:47.974123 20817 leader_election.cc:288] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef693e68c7b4405ab119641e361347a3 (127.1.253.67:44011), 88bf283162df4b86943b3958377a8880 (127.1.253.65:40447)
I20260924 16:15:47.979154 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.974564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:38203 (local address 127.1.253.67:44011)
0924 16:15:47.974707 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:47.974711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:47.974725 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:15:47.974877 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:47.974880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:47.974934 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:47.975003 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:47.975415 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.976157 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.977793 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.978428 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.978562 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.978623 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:47.978960 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 16:15:47.979026 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:15:47.979154 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.974968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:39971 (local address 127.1.253.65:40447)
0924 16:15:47.975096 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:47.975101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:47.975117 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:15:47.975230 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:47.975233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:47.975286 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:47.975375 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:47.975702 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.976369 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.977793 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.978428 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.978562 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.978633 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:47.978954 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 16:15:47.979025 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:15:47.979188 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.974459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:44011 (local address 127.1.253.66:38203)
0924 16:15:47.974595 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:47.974614 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:47.974729 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:47.975144 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:47.975147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:47.975169 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:47.975288 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.975295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.976265 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.976268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.977650 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.977662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.978614 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.978618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.978803 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.978837 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:15:47.978939 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 16:15:47.979027 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":131}
I20260924 16:15:47.979197 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.974715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40447 (local address 127.1.253.66:39971)
0924 16:15:47.975025 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:47.975039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:47.975129 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:47.975406 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:47.975409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:47.975427 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:47.975575 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.975582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.976483 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.976486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.977680 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:47.977689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:47.978614 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:47.978618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:47.978791 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:47.978846 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 16:15:47.978929 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 16:15:47.979027 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":90,"thread_start_us":89,"threads_started":1}
I20260924 16:15:47.980046 20472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13d07a594334fc991c0c4771e33d03d" candidate_uuid: "d321970c59d0449c905ccf4ef0a798c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88bf283162df4b86943b3958377a8880" is_pre_election: true
I20260924 16:15:47.980046 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13d07a594334fc991c0c4771e33d03d" candidate_uuid: "d321970c59d0449c905ccf4ef0a798c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef693e68c7b4405ab119641e361347a3" is_pre_election: true
I20260924 16:15:47.980377 20472 raft_consensus.cc:2454] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d321970c59d0449c905ccf4ef0a798c3 in term 0.
I20260924 16:15:47.980377 20748 raft_consensus.cc:2454] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d321970c59d0449c905ccf4ef0a798c3 in term 0.
I20260924 16:15:47.980901 20542 leader_election.cc:302] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88bf283162df4b86943b3958377a8880, d321970c59d0449c905ccf4ef0a798c3; no voters:
I20260924 16:15:47.981158 20817 raft_consensus.cc:2781] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:15:47.981263 20817 raft_consensus.cc:482] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:15:47.981321 20817 raft_consensus.cc:3037] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:47.982550 20817 raft_consensus.cc:504] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d321970c59d0449c905ccf4ef0a798c3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42187 } } peers { permanent_uuid: "ef693e68c7b4405ab119641e361347a3" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44011 } } peers { permanent_uuid: "88bf283162df4b86943b3958377a8880" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40447 } }
I20260924 16:15:47.982903 20817 leader_election.cc:288] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [CANDIDATE]: Term 1 election: Requested vote from peers ef693e68c7b4405ab119641e361347a3 (127.1.253.67:44011), 88bf283162df4b86943b3958377a8880 (127.1.253.65:40447)
I20260924 16:15:47.983428 20472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13d07a594334fc991c0c4771e33d03d" candidate_uuid: "d321970c59d0449c905ccf4ef0a798c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88bf283162df4b86943b3958377a8880"
I20260924 16:15:47.983397 20748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13d07a594334fc991c0c4771e33d03d" candidate_uuid: "d321970c59d0449c905ccf4ef0a798c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef693e68c7b4405ab119641e361347a3"
I20260924 16:15:47.983554 20472 raft_consensus.cc:3037] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:47.983554 20748 raft_consensus.cc:3037] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:47.984896 20748 raft_consensus.cc:2454] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d321970c59d0449c905ccf4ef0a798c3 in term 1.
I20260924 16:15:47.984891 20472 raft_consensus.cc:2454] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d321970c59d0449c905ccf4ef0a798c3 in term 1.
I20260924 16:15:47.985224 20542 leader_election.cc:302] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88bf283162df4b86943b3958377a8880, d321970c59d0449c905ccf4ef0a798c3; no voters:
I20260924 16:15:47.985391 20817 raft_consensus.cc:2781] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:47.985654 20817 raft_consensus.cc:686] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 1 LEADER]: Becoming Leader. State: Replica: d321970c59d0449c905ccf4ef0a798c3, State: Running, Role: LEADER
I20260924 16:15:47.986049 20817 consensus_queue.cc:237] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d321970c59d0449c905ccf4ef0a798c3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42187 } } peers { permanent_uuid: "ef693e68c7b4405ab119641e361347a3" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44011 } } peers { permanent_uuid: "88bf283162df4b86943b3958377a8880" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40447 } }
I20260924 16:15:47.989624 20321 catalog_manager.cc:5495] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d321970c59d0449c905ccf4ef0a798c3 (127.1.253.66). New cstate: current_term: 1 leader_uuid: "d321970c59d0449c905ccf4ef0a798c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d321970c59d0449c905ccf4ef0a798c3" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef693e68c7b4405ab119641e361347a3" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88bf283162df4b86943b3958377a8880" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40447 } health_report { overall_health: UNKNOWN } } }
W20260924 16:15:47.993120 20659 tablet.cc:2369] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:15:48.017531 20520 tablet.cc:2369] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:15:48.021334 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.017792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47542 (local address 127.1.253.66:42187)
0924 16:15:48.017919 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:48.017923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:48.018020 (+ 97us) server_negotiation.cc:407] Connection header received
0924 16:15:48.018190 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:48.018193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:48.018273 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:48.018362 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:15:48.018622 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.019111 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.019960 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.020165 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.020240 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.020519 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:15:48.020938 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:15:48.021048 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:48.021169 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 16:15:48.021205 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 16:15:48.027563 20748 raft_consensus.cc:1261] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3 [term 1 FOLLOWER]: Refusing update from remote peer d321970c59d0449c905ccf4ef0a798c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:48.027563 20472 raft_consensus.cc:1261] T b13d07a594334fc991c0c4771e33d03d P 88bf283162df4b86943b3958377a8880 [term 1 FOLLOWER]: Refusing update from remote peer d321970c59d0449c905ccf4ef0a798c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:48.028316 20817 consensus_queue.cc:1035] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88bf283162df4b86943b3958377a8880" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 40447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:48.028545 20823 consensus_queue.cc:1035] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef693e68c7b4405ab119641e361347a3" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 44011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:48.037734 20827 mvcc.cc:205] Tried to move back new op lower bound from 7332931780712726528 to 7332931780554117120. Current Snapshot: MvccSnapshot[applied={T|T < 7332931780712726528}]
I20260924 16:15:48.045878 20321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:48.046074 20321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:48.048704 20321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 041a36d972894754bdc57a6098309d09: Sending DeleteTablet for 3 replicas of tablet b13d07a594334fc991c0c4771e33d03d
I20260924 16:15:48.049669 20451 tablet_service.cc:1514] Processing DeleteTablet for tablet b13d07a594334fc991c0c4771e33d03d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:48 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:37374
I20260924 16:15:48.049701 20728 tablet_service.cc:1514] Processing DeleteTablet for tablet b13d07a594334fc991c0c4771e33d03d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:48 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:47454
I20260924 16:15:48.050225 20590 tablet_service.cc:1514] Processing DeleteTablet for tablet b13d07a594334fc991c0c4771e33d03d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:48 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:47538
I20260924 16:15:48.050349 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 20385
I20260924 16:15:48.052868 20839 tablet_replica.cc:330] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3: stopping tablet replica
I20260924 16:15:48.053236 20839 raft_consensus.cc:2229] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:15:48.053685 20839 raft_consensus.cc:2258] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:48.054097 20838 tablet_replica.cc:330] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3: stopping tablet replica
I20260924 16:15:48.054440 20838 raft_consensus.cc:2229] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:48.054774 20838 raft_consensus.cc:2258] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:48.056391 20839 ts_tablet_manager.cc:1905] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:48.056571 20838 ts_tablet_manager.cc:1905] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 16:15:48.059283 20306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:40447 (error 108)
I20260924 16:15:48.059520 20838 ts_tablet_manager.cc:1918] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:48.059685 20838 log.cc:1196] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/b13d07a594334fc991c0c4771e33d03d
I20260924 16:15:48.059713 20839 ts_tablet_manager.cc:1918] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:48.059890 20839 log.cc:1196] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/b13d07a594334fc991c0c4771e33d03d
I20260924 16:15:48.060118 20838 ts_tablet_manager.cc:1939] T b13d07a594334fc991c0c4771e33d03d P ef693e68c7b4405ab119641e361347a3: Deleting consensus metadata
I20260924 16:15:48.060138 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 20525
I20260924 16:15:48.060282 20839 ts_tablet_manager.cc:1939] T b13d07a594334fc991c0c4771e33d03d P d321970c59d0449c905ccf4ef0a798c3: Deleting consensus metadata
I20260924 16:15:48.061815 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.061002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:40447 (local address 127.0.0.1:37380)
0924 16:15:48.061641 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:48.061701 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558}
W20260924 16:15:48.062605 20306 catalog_manager.cc:4584] TS 88bf283162df4b86943b3958377a8880 (127.1.253.65:40447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b13d07a594334fc991c0c4771e33d03d: Network error: Client connection negotiation failed: client connection to 127.1.253.65:40447: connect: Connection refused (error 111)
I20260924 16:15:48.062821 20306 catalog_manager.cc:4841] TS ef693e68c7b4405ab119641e361347a3 (127.1.253.67:44011): tablet b13d07a594334fc991c0c4771e33d03d (table test-table [id=fcada004f1f24dbb8c9db4274297e3d9]) successfully deleted
I20260924 16:15:48.067672 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 20663
I20260924 16:15:48.068699 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.067896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:42187 (local address 127.0.0.1:47550)
0924 16:15:48.068523 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:48.068590 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.66:42187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510}
I20260924 16:15:48.073869 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 20289
2026-09-24T16:15:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2527 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T16:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:48Z Disabled control of system clock
I20260924 16:15:48.106015 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:41133
--webserver_interface=127.1.253.126
--webserver_port=0
--builtin_ntp_servers=127.1.253.84:45797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:41133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:15:48.223071 20844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:48.223465 20844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:48.223537 20844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:48.228044 20844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:48.228116 20844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:48.228132 20844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:48.228148 20844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:48.228164 20844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:48.233227 20844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:41133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:41133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:48.234465 20844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:48.235936 20844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:48.245127 20849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:48.245131 20850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:48.245131 20852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:48.245183 20844 server_base.cc:1034] running on GCE node
I20260924 16:15:48.245992 20844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:48.247238 20844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:48.248430 20844 hybrid_clock.cc:648] HybridClock initialized: now 1790266548248416 us; error 39 us; skew 500 ppm
I20260924 16:15:48.250538 20844 webserver.cc:460] Webserver started at http://127.1.253.126:46385/ using document root <none> and password file <none>
I20260924 16:15:48.251159 20844 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:48.251258 20844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:48.251523 20844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:48.253732 20844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "dcc7356d2b5d45a6a2bb94a896f0df54"
format_stamp: "Formatted at 2026-09-24 16:15:48 on dist-test-slave-6wvm"
server_key: "37b840624a576bbe5f1478d3b1c89f57"
server_key_iv: "b7cabed5d66937e4d972be956fedc283"
server_key_version: "encryptionkey@0"
I20260924 16:15:48.254385 20844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "dcc7356d2b5d45a6a2bb94a896f0df54"
format_stamp: "Formatted at 2026-09-24 16:15:48 on dist-test-slave-6wvm"
server_key: "37b840624a576bbe5f1478d3b1c89f57"
server_key_iv: "b7cabed5d66937e4d972be956fedc283"
server_key_version: "encryptionkey@0"
I20260924 16:15:48.258170 20844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:48.260780 20858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:48.262123 20844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:15:48.262334 20844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "dcc7356d2b5d45a6a2bb94a896f0df54"
format_stamp: "Formatted at 2026-09-24 16:15:48 on dist-test-slave-6wvm"
server_key: "37b840624a576bbe5f1478d3b1c89f57"
server_key_iv: "b7cabed5d66937e4d972be956fedc283"
server_key_version: "encryptionkey@0"
I20260924 16:15:48.262456 20844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:48.272305 20844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:48.273016 20844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:48.273228 20844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:48.280874 20844 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:41133
I20260924 16:15:48.280906 20912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:41133 every 8 connection(s)
I20260924 16:15:48.281904 20844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:48.282516 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 20844
I20260924 16:15:48.282655 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:15:48.282965 2037 external_mini_cluster.cc:1428] Setting key 1d926a48607d4194753e52f99be2b57d
I20260924 16:15:48.285756 20913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:48.292177 20913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54: Bootstrap starting.
I20260924 16:15:48.292819 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.284499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.1.253.126:41133)
0924 16:15:48.285184 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:48.285201 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:48.285237 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:15:48.285948 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:48.285969 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:48.286473 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:48.286837 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:48.287309 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.288289 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.289218 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.289717 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.289900 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.290873 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:48.290878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:48.290892 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:48.291311 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:48.291373 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:48.291394 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:48.291509 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:48.291648 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 16:15:48.291933 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":152,"threads_started":1}
I20260924 16:15:48.294729 20913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:48.295660 20913 log.cc:824] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:48.297838 20913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54: No bootstrap required, opened a new log
I20260924 16:15:48.301787 20913 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc7356d2b5d45a6a2bb94a896f0df54" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41133 } }
I20260924 16:15:48.302070 20913 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:48.302158 20913 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc7356d2b5d45a6a2bb94a896f0df54, State: Initialized, Role: FOLLOWER
I20260924 16:15:48.302654 20913 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc7356d2b5d45a6a2bb94a896f0df54" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41133 } }
I20260924 16:15:48.302814 20913 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:48.302907 20913 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:48.303040 20913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:48.304035 20913 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc7356d2b5d45a6a2bb94a896f0df54" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41133 } }
I20260924 16:15:48.304394 20913 leader_election.cc:302] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcc7356d2b5d45a6a2bb94a896f0df54; no voters:
I20260924 16:15:48.304739 20913 leader_election.cc:288] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:48.304880 20918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:48.305150 20918 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [term 1 LEADER]: Becoming Leader. State: Replica: dcc7356d2b5d45a6a2bb94a896f0df54, State: Running, Role: LEADER
I20260924 16:15:48.305546 20918 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc7356d2b5d45a6a2bb94a896f0df54" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41133 } }
I20260924 16:15:48.306116 20913 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:48.307614 20919 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcc7356d2b5d45a6a2bb94a896f0df54. Latest consensus state: current_term: 1 leader_uuid: "dcc7356d2b5d45a6a2bb94a896f0df54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc7356d2b5d45a6a2bb94a896f0df54" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41133 } } }
I20260924 16:15:48.307741 20919 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:48.307972 20920 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcc7356d2b5d45a6a2bb94a896f0df54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc7356d2b5d45a6a2bb94a896f0df54" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41133 } } }
I20260924 16:15:48.308070 20920 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:48.308159 20928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:48.311604 20928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:48.318032 20928 catalog_manager.cc:1347] Generated new cluster ID: b1f8b5f096a9420dba260d0f458d5b6a
I20260924 16:15:48.318215 20928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:48.336632 20928 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:48.337939 20928 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:48.347206 20928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54: Generated new TSK 0
I20260924 16:15:48.348090 20928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:15:48.359359 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41133
--builtin_ntp_servers=127.1.253.84:45797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:15:48.478459 20938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:48.478754 20938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:48.478826 20938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:48.482810 20938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:48.482893 20938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:48.482982 20938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:15:48.487794 20938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:48.488915 20938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:48.490413 20938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:48.500298 20943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:48.500298 20944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:48.500341 20947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:48.500627 20938 server_base.cc:1034] running on GCE node
I20260924 16:15:48.501178 20938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:48.501825 20938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:48.503024 20938 hybrid_clock.cc:648] HybridClock initialized: now 1790266548503010 us; error 26 us; skew 500 ppm
I20260924 16:15:48.504949 20938 webserver.cc:460] Webserver started at http://127.1.253.65:43027/ using document root <none> and password file <none>
I20260924 16:15:48.505581 20938 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:48.505676 20938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:48.505934 20938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:48.508301 20938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "e2b23919f3334ec284b485ee773fef89"
format_stamp: "Formatted at 2026-09-24 16:15:48 on dist-test-slave-6wvm"
server_key: "00492b6b248a80e23aa28191f442cf48"
server_key_iv: "eccd0880b0d9d1ed067172d626ef6665"
server_key_version: "encryptionkey@0"
I20260924 16:15:48.508868 20938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "e2b23919f3334ec284b485ee773fef89"
format_stamp: "Formatted at 2026-09-24 16:15:48 on dist-test-slave-6wvm"
server_key: "00492b6b248a80e23aa28191f442cf48"
server_key_iv: "eccd0880b0d9d1ed067172d626ef6665"
server_key_version: "encryptionkey@0"
I20260924 16:15:48.512758 20938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 16:15:48.515357 20953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:48.516611 20938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:15:48.516754 20938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "e2b23919f3334ec284b485ee773fef89"
format_stamp: "Formatted at 2026-09-24 16:15:48 on dist-test-slave-6wvm"
server_key: "00492b6b248a80e23aa28191f442cf48"
server_key_iv: "eccd0880b0d9d1ed067172d626ef6665"
server_key_version: "encryptionkey@0"
I20260924 16:15:48.516898 20938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:48.531183 20938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:48.531831 20938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:48.532034 20938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:48.532622 20938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:48.533597 20938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:48.533674 20938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:48.533759 20938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:48.533802 20938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:48.543752 20938 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:43289
I20260924 16:15:48.543777 21069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:43289 every 8 connection(s)
I20260924 16:15:48.544680 20938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:48.546619 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 20938
I20260924 16:15:48.546749 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:15:48.547062 2037 external_mini_cluster.cc:1428] Setting key 2a6301410ea0aac81088abbbde68e562
I20260924 16:15:48.549309 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41133
--builtin_ntp_servers=127.1.253.84:45797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:15:48.555734 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.546734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:53185 (local address 127.1.253.126:41133)
0924 16:15:48.546881 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:48.546886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:48.548177 (+ 1291us) server_negotiation.cc:407] Connection header received
0924 16:15:48.550090 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:48.550093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:48.550142 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:48.550215 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:48.551417 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.552348 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.553499 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.553921 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.554054 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.555198 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:48.555202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:48.555205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:48.555224 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:48.555242 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:48.555250 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:48.555331 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:48.555496 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 16:15:48.555577 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:15:48.556595 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.547230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:41133 (local address 127.1.253.65:53185)
0924 16:15:48.548037 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:48.548077 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:48.549844 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:48.550473 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:48.550485 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:48.550601 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:48.551279 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:48.551293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.552481 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.552484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:48.553359 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:48.553368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.554078 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.554080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:48.554214 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.554925 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:15:48.554926 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:48.554966 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:15:48.555031 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:48.555333 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:48.555346 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:48.555352 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:15:48.555496 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 16:15:48.555775 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":160,"threads_started":1}
I20260924 16:15:48.557818 21070 heartbeater.cc:347] Connected to a master server at 127.1.253.126:41133
I20260924 16:15:48.558112 21070 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:48.558833 21070 heartbeater.cc:511] Master 127.1.253.126:41133 requested a full tablet report, sending...
I20260924 16:15:48.560510 20875 ts_manager.cc:194] Registered new tserver with Master: e2b23919f3334ec284b485ee773fef89 (127.1.253.65:43289)
I20260924 16:15:48.561674 20875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.253.65:53185
W20260924 16:15:48.666370 21073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:48.666623 21073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:48.666690 21073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:48.670682 21073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:48.670732 21073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:48.670817 21073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:15:48.675877 21073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:48.677049 21073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:48.678515 21073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:15:48.690295 21073 server_base.cc:1034] running on GCE node
W20260924 16:15:48.690373 21078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:48.690402 21079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:48.690094 21081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:48.690923 21073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:48.691910 21073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:48.695174 21073 hybrid_clock.cc:648] HybridClock initialized: now 1790266548695121 us; error 36 us; skew 500 ppm
I20260924 16:15:48.697528 21073 webserver.cc:460] Webserver started at http://127.1.253.66:44375/ using document root <none> and password file <none>
I20260924 16:15:48.698312 21073 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:48.698416 21073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:48.698693 21073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:48.701524 21073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "d80304effe884cee9eafced5371101f4"
format_stamp: "Formatted at 2026-09-24 16:15:48 on dist-test-slave-6wvm"
server_key: "fa33e4520620e9f2e921206e7c539e22"
server_key_iv: "3e9a08319b2bafac5c7fa74157b88957"
server_key_version: "encryptionkey@0"
I20260924 16:15:48.702181 21073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "d80304effe884cee9eafced5371101f4"
format_stamp: "Formatted at 2026-09-24 16:15:48 on dist-test-slave-6wvm"
server_key: "fa33e4520620e9f2e921206e7c539e22"
server_key_iv: "3e9a08319b2bafac5c7fa74157b88957"
server_key_version: "encryptionkey@0"
I20260924 16:15:48.707697 21073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:15:48.711056 21087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:48.712720 21073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:48.712862 21073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "d80304effe884cee9eafced5371101f4"
format_stamp: "Formatted at 2026-09-24 16:15:48 on dist-test-slave-6wvm"
server_key: "fa33e4520620e9f2e921206e7c539e22"
server_key_iv: "3e9a08319b2bafac5c7fa74157b88957"
server_key_version: "encryptionkey@0"
I20260924 16:15:48.712996 21073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:48.732637 21073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:48.733404 21073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:48.733620 21073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:48.734336 21073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:48.735368 21073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:48.735441 21073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:48.735513 21073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:48.735565 21073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:48.744696 21073 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:39115
I20260924 16:15:48.744720 21201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:39115 every 8 connection(s)
I20260924 16:15:48.745646 21073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:48.746800 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 21073
I20260924 16:15:48.746941 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:15:48.747283 2037 external_mini_cluster.cc:1428] Setting key d019ce782c0ac3d8c30b0a445679b408
I20260924 16:15:48.749401 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41133
--builtin_ntp_servers=127.1.253.84:45797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:15:48.756604 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.747761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:36637 (local address 127.1.253.126:41133)
0924 16:15:48.747942 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:48.747947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:48.749201 (+ 1254us) server_negotiation.cc:407] Connection header received
0924 16:15:48.750703 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:48.750708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:48.750780 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:48.750881 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:48.752174 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.753011 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.754298 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.754727 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.754860 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.755981 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:48.755987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:48.755993 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:48.756012 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:48.756040 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:48.756056 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:48.756152 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:48.756314 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 16:15:48.756396 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:15:48.757367 21204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.748278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:41133 (local address 127.1.253.66:36637)
0924 16:15:48.749030 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:48.749086 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:48.750476 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:48.751102 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:48.751115 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:48.751264 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:48.752017 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:48.752039 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.753144 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.753147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:48.754158 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:48.754168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.754879 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.754882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:48.755027 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.755702 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:15:48.755704 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:48.755745 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:15:48.755812 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:48.756165 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:48.756172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:48.756177 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:15:48.756270 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:15:48.756492 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":151,"threads_started":1}
I20260924 16:15:48.758723 21202 heartbeater.cc:347] Connected to a master server at 127.1.253.126:41133
I20260924 16:15:48.759023 21202 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:48.759739 21202 heartbeater.cc:511] Master 127.1.253.126:41133 requested a full tablet report, sending...
I20260924 16:15:48.761011 20875 ts_manager.cc:194] Registered new tserver with Master: d80304effe884cee9eafced5371101f4 (127.1.253.66:39115)
I20260924 16:15:48.761624 20875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.253.66:36637
W20260924 16:15:48.870540 21206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:48.870826 21206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:48.870893 21206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:48.874955 21206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:48.875021 21206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:48.875078 21206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:15:48.880080 21206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:48.881326 21206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:48.882915 21206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:48.893208 21212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:48.893235 21214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:48.893389 21206 server_base.cc:1034] running on GCE node
W20260924 16:15:48.893219 21211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:48.894039 21206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:48.894768 21206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:48.895951 21206 hybrid_clock.cc:648] HybridClock initialized: now 1790266548895912 us; error 30 us; skew 500 ppm
I20260924 16:15:48.897979 21206 webserver.cc:460] Webserver started at http://127.1.253.67:40065/ using document root <none> and password file <none>
I20260924 16:15:48.898653 21206 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:48.898746 21206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:48.898999 21206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:48.901161 21206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "24404c724f8e41f2969576ab6a4386a6"
format_stamp: "Formatted at 2026-09-24 16:15:48 on dist-test-slave-6wvm"
server_key: "3a54dd213441b7da486b34bfe5f86546"
server_key_iv: "e4c15c69a7113fa6f01194815b606d32"
server_key_version: "encryptionkey@0"
I20260924 16:15:48.901759 21206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "24404c724f8e41f2969576ab6a4386a6"
format_stamp: "Formatted at 2026-09-24 16:15:48 on dist-test-slave-6wvm"
server_key: "3a54dd213441b7da486b34bfe5f86546"
server_key_iv: "e4c15c69a7113fa6f01194815b606d32"
server_key_version: "encryptionkey@0"
I20260924 16:15:48.905705 21206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:15:48.908530 21220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:48.909790 21206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:15:48.909924 21206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "24404c724f8e41f2969576ab6a4386a6"
format_stamp: "Formatted at 2026-09-24 16:15:48 on dist-test-slave-6wvm"
server_key: "3a54dd213441b7da486b34bfe5f86546"
server_key_iv: "e4c15c69a7113fa6f01194815b606d32"
server_key_version: "encryptionkey@0"
I20260924 16:15:48.910045 21206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:48.929024 21206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:48.929738 21206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:48.929948 21206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:48.930622 21206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:48.931595 21206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:48.931667 21206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:48.931744 21206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:48.931790 21206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:48.940551 21206 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:38453
I20260924 16:15:48.940578 21335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:38453 every 8 connection(s)
I20260924 16:15:48.941488 21206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:48.946935 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 21206
I20260924 16:15:48.947023 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:15:48.947296 2037 external_mini_cluster.cc:1428] Setting key 107ef70b1e6b9df062411e95cfd24f6c
I20260924 16:15:48.951071 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.943412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:50325 (local address 127.1.253.126:41133)
0924 16:15:48.943561 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:48.943565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:48.944583 (+ 1018us) server_negotiation.cc:407] Connection header received
0924 16:15:48.946013 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:48.946016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:48.946073 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:48.946180 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:48.947151 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.947812 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.949120 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.949347 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.949457 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.950578 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:48.950582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:48.950585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:48.950597 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:48.950624 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:48.950632 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:48.950710 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:48.950865 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 16:15:48.950924 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:15:48.951881 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.943811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:41133 (local address 127.1.253.67:50325)
0924 16:15:48.944439 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:48.944481 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:48.945792 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:48.946377 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:48.946384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:48.946475 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:48.946943 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:48.946954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.947954 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.947958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:48.948988 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:48.948998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.949501 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.949506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:48.949669 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.950307 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:15:48.950309 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:48.950350 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:15:48.950414 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:48.950712 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:48.950723 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:48.950732 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:15:48.950864 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 16:15:48.951146 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":104,"threads_started":1}
I20260924 16:15:48.952965 21336 heartbeater.cc:347] Connected to a master server at 127.1.253.126:41133
I20260924 16:15:48.953210 21336 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:48.953727 21336 heartbeater.cc:511] Master 127.1.253.126:41133 requested a full tablet report, sending...
I20260924 16:15:48.954761 20875 ts_manager.cc:194] Registered new tserver with Master: 24404c724f8e41f2969576ab6a4386a6 (127.1.253.67:38453)
I20260924 16:15:48.955338 20875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.253.67:50325
I20260924 16:15:48.961146 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:15:48.966634 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.963084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.1.253.126:41133)
0924 16:15:48.963211 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:48.963215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:48.963399 (+ 184us) server_negotiation.cc:407] Connection header received
0924 16:15:48.963511 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:48.963513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:48.963552 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:48.963613 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:48.963837 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.964327 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.965120 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.965320 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.965396 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.966162 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:48.966165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:48.966167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:48.966179 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:48.966198 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:48.966207 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:48.966378 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:48.966493 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 16:15:48.966523 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:15:48.969515 20875 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:15:48.972666 20875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:15:48.988000 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.981616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:38453 (local address 127.0.0.1:57304)
0924 16:15:48.981991 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:48.982044 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:48.982194 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:48.982947 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:48.982952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:48.982975 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:48.983180 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:48.983192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.984751 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.984757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:48.986428 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:48.986441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.987525 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.987528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:48.987633 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.987678 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 16:15:48.987764 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 16:15:48.987881 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":18,"thread_start_us":84,"threads_started":1}
I20260924 16:15:48.988581 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.983608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:43289 (local address 127.0.0.1:43148)
0924 16:15:48.984056 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:48.984069 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:48.984173 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:48.984842 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:48.984845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:48.984865 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:48.984958 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:48.984964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.986023 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.986028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:48.987255 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:48.987267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.988228 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.988232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:48.988312 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.988334 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:15:48.988396 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 16:15:48.988456 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":35,"thread_start_us":92,"threads_started":1}
I20260924 16:15:48.988785 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.981788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.1.253.67:38453)
0924 16:15:48.982354 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:48.982370 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:48.982398 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:15:48.982480 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:48.982485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:48.982711 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:48.983494 (+ 783us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:48.983639 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.984633 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.986578 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.987402 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.988238 (+ 836us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.988459 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:48.988511 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 16:15:48.988612 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":182,"threads_started":1}
I20260924 16:15:48.989114 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.984020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.1.253.65:43289)
0924 16:15:48.984420 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:48.984430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:48.984454 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:15:48.984527 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:48.984531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:48.984705 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:48.984851 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:48.985079 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.985906 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.987380 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.987979 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.988105 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.988790 (+ 685us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:48.988842 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 16:15:48.988936 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":140,"threads_started":1}
I20260924 16:15:48.989382 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.981896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:39115 (local address 127.0.0.1:44900)
0924 16:15:48.982762 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:48.982802 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:48.982940 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:48.983921 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:48.983924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:48.983944 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:48.984046 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:48.984053 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.985419 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.985422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:48.987684 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:48.987691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.988974 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.988978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:48.989069 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.989091 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:15:48.989176 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 16:15:48.989252 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260924 16:15:48.989761 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.982508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.1.253.66:39115)
0924 16:15:48.983267 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:48.983281 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:48.983306 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:15:48.983383 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:48.983388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:48.983609 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:48.983766 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:48.984190 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.985291 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.987819 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:48.988615 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:48.988738 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:48.988911 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:48.989361 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 16:15:48.989609 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":134,"threads_started":1}
I20260924 16:15:48.991919 21269 tablet_service.cc:1467] Processing CreateTablet for tablet 6012852fde484986ae8b67b2ffc2e8cf (DEFAULT_TABLE table=test-table [id=c710e19b9cb646ea9a5831f2fce334d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:48.992033 21003 tablet_service.cc:1467] Processing CreateTablet for tablet 6012852fde484986ae8b67b2ffc2e8cf (DEFAULT_TABLE table=test-table [id=c710e19b9cb646ea9a5831f2fce334d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:48.992269 21135 tablet_service.cc:1467] Processing CreateTablet for tablet 6012852fde484986ae8b67b2ffc2e8cf (DEFAULT_TABLE table=test-table [id=c710e19b9cb646ea9a5831f2fce334d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:48.992859 21269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6012852fde484986ae8b67b2ffc2e8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:48.993072 21135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6012852fde484986ae8b67b2ffc2e8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:48.993072 21003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6012852fde484986ae8b67b2ffc2e8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:49.000221 21353 tablet_bootstrap.cc:492] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6: Bootstrap starting.
I20260924 16:15:49.000845 21354 tablet_bootstrap.cc:492] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89: Bootstrap starting.
I20260924 16:15:49.002405 21353 tablet_bootstrap.cc:654] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:49.002600 21355 tablet_bootstrap.cc:492] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4: Bootstrap starting.
I20260924 16:15:49.002707 21354 tablet_bootstrap.cc:654] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:49.003235 21353 log.cc:824] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:49.003537 21354 log.cc:824] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:49.004738 21355 tablet_bootstrap.cc:654] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:49.005401 21353 tablet_bootstrap.cc:492] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6: No bootstrap required, opened a new log
I20260924 16:15:49.005496 21355 log.cc:824] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:49.005595 21353 ts_tablet_manager.cc:1397] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260924 16:15:49.005980 21354 tablet_bootstrap.cc:492] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89: No bootstrap required, opened a new log
I20260924 16:15:49.006189 21354 ts_tablet_manager.cc:1397] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 16:15:49.008348 21355 tablet_bootstrap.cc:492] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4: No bootstrap required, opened a new log
I20260924 16:15:49.008562 21355 ts_tablet_manager.cc:1397] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 16:15:49.009498 21354 raft_consensus.cc:348] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24404c724f8e41f2969576ab6a4386a6" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38453 } } peers { permanent_uuid: "e2b23919f3334ec284b485ee773fef89" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 43289 } } peers { permanent_uuid: "d80304effe884cee9eafced5371101f4" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 39115 } }
I20260924 16:15:49.009737 21354 raft_consensus.cc:374] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:49.009788 21354 raft_consensus.cc:729] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b23919f3334ec284b485ee773fef89, State: Initialized, Role: FOLLOWER
I20260924 16:15:49.010296 21354 consensus_queue.cc:260] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24404c724f8e41f2969576ab6a4386a6" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38453 } } peers { permanent_uuid: "e2b23919f3334ec284b485ee773fef89" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 43289 } } peers { permanent_uuid: "d80304effe884cee9eafced5371101f4" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 39115 } }
I20260924 16:15:49.010187 21353 raft_consensus.cc:348] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24404c724f8e41f2969576ab6a4386a6" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38453 } } peers { permanent_uuid: "e2b23919f3334ec284b485ee773fef89" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 43289 } } peers { permanent_uuid: "d80304effe884cee9eafced5371101f4" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 39115 } }
I20260924 16:15:49.010475 21353 raft_consensus.cc:374] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:49.010536 21353 raft_consensus.cc:729] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24404c724f8e41f2969576ab6a4386a6, State: Initialized, Role: FOLLOWER
I20260924 16:15:49.011015 21353 consensus_queue.cc:260] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24404c724f8e41f2969576ab6a4386a6" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38453 } } peers { permanent_uuid: "e2b23919f3334ec284b485ee773fef89" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 43289 } } peers { permanent_uuid: "d80304effe884cee9eafced5371101f4" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 39115 } }
I20260924 16:15:49.011301 21354 ts_tablet_manager.cc:1428] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260924 16:15:49.011710 21070 heartbeater.cc:503] Master 127.1.253.126:41133 was elected leader, sending a full tablet report...
I20260924 16:15:49.012013 21353 ts_tablet_manager.cc:1428] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 16:15:49.012468 21355 raft_consensus.cc:348] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24404c724f8e41f2969576ab6a4386a6" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38453 } } peers { permanent_uuid: "e2b23919f3334ec284b485ee773fef89" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 43289 } } peers { permanent_uuid: "d80304effe884cee9eafced5371101f4" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 39115 } }
I20260924 16:15:49.012765 21355 raft_consensus.cc:374] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:49.012820 21355 raft_consensus.cc:729] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80304effe884cee9eafced5371101f4, State: Initialized, Role: FOLLOWER
I20260924 16:15:49.012964 21336 heartbeater.cc:503] Master 127.1.253.126:41133 was elected leader, sending a full tablet report...
I20260924 16:15:49.013252 21355 consensus_queue.cc:260] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24404c724f8e41f2969576ab6a4386a6" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38453 } } peers { permanent_uuid: "e2b23919f3334ec284b485ee773fef89" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 43289 } } peers { permanent_uuid: "d80304effe884cee9eafced5371101f4" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 39115 } }
I20260924 16:15:49.014317 21355 ts_tablet_manager.cc:1428] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260924 16:15:49.014644 21202 heartbeater.cc:503] Master 127.1.253.126:41133 was elected leader, sending a full tablet report...
I20260924 16:15:49.038950 21360 raft_consensus.cc:482] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:49.039186 21360 raft_consensus.cc:504] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24404c724f8e41f2969576ab6a4386a6" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38453 } } peers { permanent_uuid: "e2b23919f3334ec284b485ee773fef89" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 43289 } } peers { permanent_uuid: "d80304effe884cee9eafced5371101f4" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 39115 } }
I20260924 16:15:49.040374 21360 leader_election.cc:288] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2b23919f3334ec284b485ee773fef89 (127.1.253.65:43289), d80304effe884cee9eafced5371101f4 (127.1.253.66:39115)
I20260924 16:15:49.045600 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:49.040732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:49619 (local address 127.1.253.66:39115)
0924 16:15:49.040882 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:49.040887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:49.041127 (+ 240us) server_negotiation.cc:407] Connection header received
0924 16:15:49.041202 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:49.041205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:49.041255 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:49.041330 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:49.041768 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:49.042605 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:49.044420 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:49.044991 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:49.045107 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:49.045162 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:49.045385 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 16:15:49.045463 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:15:49.045610 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:49.040725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:52775 (local address 127.1.253.65:43289)
0924 16:15:49.041007 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:49.041010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:49.041024 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:15:49.041166 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:49.041171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:49.041227 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:49.041315 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:49.041768 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:49.042600 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:49.044306 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:49.044917 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:49.045058 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:49.045115 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:49.045385 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 16:15:49.045463 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260924 16:15:49.045634 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:49.040605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:43289 (local address 127.1.253.67:52775)
0924 16:15:49.040908 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:49.040924 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:49.041021 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:49.041355 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:49.041358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:49.041398 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:49.041636 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:49.041645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:49.042729 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:49.042733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:49.044178 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:49.044185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:49.045063 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:49.045069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:49.045234 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:49.045276 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 16:15:49.045350 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 16:15:49.045467 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":270,"spinlock_wait_cycles":7552}
I20260924 16:15:49.045650 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:49.040624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:39115 (local address 127.1.253.67:49619)
0924 16:15:49.041025 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:49.041036 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:49.041122 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:49.041355 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:49.041357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:49.041397 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:49.041644 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:49.041650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:49.042729 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:49.042733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:49.044173 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:49.044185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:49.045088 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:49.045091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:49.045218 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:49.045270 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 16:15:49.045350 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 16:15:49.045468 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":245,"spinlock_wait_cycles":30464,"thread_start_us":123,"threads_started":1}
W20260924 16:15:49.045970 21071 tablet.cc:2369] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:15:49.046358 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6012852fde484986ae8b67b2ffc2e8cf" candidate_uuid: "24404c724f8e41f2969576ab6a4386a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d80304effe884cee9eafced5371101f4" is_pre_election: true
I20260924 16:15:49.046355 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6012852fde484986ae8b67b2ffc2e8cf" candidate_uuid: "24404c724f8e41f2969576ab6a4386a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2b23919f3334ec284b485ee773fef89" is_pre_election: true
I20260924 16:15:49.046679 21024 raft_consensus.cc:2454] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24404c724f8e41f2969576ab6a4386a6 in term 0.
I20260924 16:15:49.046679 21156 raft_consensus.cc:2454] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24404c724f8e41f2969576ab6a4386a6 in term 0.
I20260924 16:15:49.047251 21225 leader_election.cc:302] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 24404c724f8e41f2969576ab6a4386a6, d80304effe884cee9eafced5371101f4, e2b23919f3334ec284b485ee773fef89; no voters:
I20260924 16:15:49.047571 21360 raft_consensus.cc:2781] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:15:49.047667 21360 raft_consensus.cc:482] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:15:49.047729 21360 raft_consensus.cc:3037] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:49.048781 21360 raft_consensus.cc:504] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24404c724f8e41f2969576ab6a4386a6" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38453 } } peers { permanent_uuid: "e2b23919f3334ec284b485ee773fef89" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 43289 } } peers { permanent_uuid: "d80304effe884cee9eafced5371101f4" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 39115 } }
I20260924 16:15:49.049151 21360 leader_election.cc:288] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [CANDIDATE]: Term 1 election: Requested vote from peers e2b23919f3334ec284b485ee773fef89 (127.1.253.65:43289), d80304effe884cee9eafced5371101f4 (127.1.253.66:39115)
I20260924 16:15:49.049542 21024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6012852fde484986ae8b67b2ffc2e8cf" candidate_uuid: "24404c724f8e41f2969576ab6a4386a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2b23919f3334ec284b485ee773fef89"
I20260924 16:15:49.049688 21024 raft_consensus.cc:3037] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:49.049736 21156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6012852fde484986ae8b67b2ffc2e8cf" candidate_uuid: "24404c724f8e41f2969576ab6a4386a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d80304effe884cee9eafced5371101f4"
I20260924 16:15:49.049896 21156 raft_consensus.cc:3037] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:49.050935 21024 raft_consensus.cc:2454] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24404c724f8e41f2969576ab6a4386a6 in term 1.
I20260924 16:15:49.051056 21156 raft_consensus.cc:2454] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24404c724f8e41f2969576ab6a4386a6 in term 1.
I20260924 16:15:49.051367 21223 leader_election.cc:302] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24404c724f8e41f2969576ab6a4386a6, e2b23919f3334ec284b485ee773fef89; no voters:
I20260924 16:15:49.051553 21360 raft_consensus.cc:2781] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:49.051738 21360 raft_consensus.cc:686] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 1 LEADER]: Becoming Leader. State: Replica: 24404c724f8e41f2969576ab6a4386a6, State: Running, Role: LEADER
I20260924 16:15:49.052063 21360 consensus_queue.cc:237] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24404c724f8e41f2969576ab6a4386a6" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38453 } } peers { permanent_uuid: "e2b23919f3334ec284b485ee773fef89" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 43289 } } peers { permanent_uuid: "d80304effe884cee9eafced5371101f4" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 39115 } }
I20260924 16:15:49.055099 20875 catalog_manager.cc:5495] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24404c724f8e41f2969576ab6a4386a6 (127.1.253.67). New cstate: current_term: 1 leader_uuid: "24404c724f8e41f2969576ab6a4386a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24404c724f8e41f2969576ab6a4386a6" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2b23919f3334ec284b485ee773fef89" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 43289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d80304effe884cee9eafced5371101f4" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 39115 } health_report { overall_health: UNKNOWN } } }
I20260924 16:15:49.068698 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:49.065056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.1.253.67:38453)
0924 16:15:49.065195 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:49.065200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:49.065214 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:15:49.065360 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:49.065362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:49.065418 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:49.065532 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:15:49.065781 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:49.066339 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:49.067220 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:49.067438 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:49.067546 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:49.067865 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:15:49.068266 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:15:49.068429 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:49.068520 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 16:15:49.068555 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:15:49.074728 21156 raft_consensus.cc:1261] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4 [term 1 FOLLOWER]: Refusing update from remote peer 24404c724f8e41f2969576ab6a4386a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:49.074728 21024 raft_consensus.cc:1261] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89 [term 1 FOLLOWER]: Refusing update from remote peer 24404c724f8e41f2969576ab6a4386a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:49.075522 21363 consensus_queue.cc:1035] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2b23919f3334ec284b485ee773fef89" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 43289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:49.075708 21360 consensus_queue.cc:1035] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d80304effe884cee9eafced5371101f4" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 39115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:49.084317 21367 mvcc.cc:205] Tried to move back new op lower bound from 7332931785002631168 to 7332931784919998464. Current Snapshot: MvccSnapshot[applied={T|T < 7332931785002631168}]
I20260924 16:15:49.085778 21369 mvcc.cc:205] Tried to move back new op lower bound from 7332931785002631168 to 7332931784919998464. Current Snapshot: MvccSnapshot[applied={T|T < 7332931785002631168}]
I20260924 16:15:49.092023 20875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:49.092180 20875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:49.094791 20875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcc7356d2b5d45a6a2bb94a896f0df54: Sending DeleteTablet for 3 replicas of tablet 6012852fde484986ae8b67b2ffc2e8cf
I20260924 16:15:49.095691 21269 tablet_service.cc:1514] Processing DeleteTablet for tablet 6012852fde484986ae8b67b2ffc2e8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:49 UTC) from {username='slave'} at 127.0.0.1:57304
I20260924 16:15:49.095705 21135 tablet_service.cc:1514] Processing DeleteTablet for tablet 6012852fde484986ae8b67b2ffc2e8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:49 UTC) from {username='slave'} at 127.0.0.1:44900
I20260924 16:15:49.095836 21003 tablet_service.cc:1514] Processing DeleteTablet for tablet 6012852fde484986ae8b67b2ffc2e8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:49 UTC) from {username='slave'} at 127.0.0.1:43148
I20260924 16:15:49.096390 21377 tablet_replica.cc:330] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89: stopping tablet replica
I20260924 16:15:49.096763 21377 raft_consensus.cc:2229] T 6012852fde484986ae8b67b2ffc2e8cf P e2b23919f3334ec284b485ee773fef89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:49.096798 21376 tablet_replica.cc:330] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4: stopping tablet replica
I20260924 16:15:49.096925 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 20938
I20260924 16:15:49.097656 21376 raft_consensus.cc:2229] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:49.097980 21376 raft_consensus.cc:2258] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:49.098384 21375 tablet_replica.cc:330] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6: stopping tablet replica
I20260924 16:15:49.098673 21375 raft_consensus.cc:2229] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:15:49.099030 21375 raft_consensus.cc:2258] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:49.099682 21376 ts_tablet_manager.cc:1905] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:49.100721 21375 ts_tablet_manager.cc:1905] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:49.102746 21376 ts_tablet_manager.cc:1918] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:49.102850 21376 log.cc:1196] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/6012852fde484986ae8b67b2ffc2e8cf
I20260924 16:15:49.103185 21376 ts_tablet_manager.cc:1939] T 6012852fde484986ae8b67b2ffc2e8cf P d80304effe884cee9eafced5371101f4: Deleting consensus metadata
I20260924 16:15:49.103624 21375 ts_tablet_manager.cc:1918] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:49.103721 21375 log.cc:1196] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/6012852fde484986ae8b67b2ffc2e8cf
I20260924 16:15:49.104036 21375 ts_tablet_manager.cc:1939] T 6012852fde484986ae8b67b2ffc2e8cf P 24404c724f8e41f2969576ab6a4386a6: Deleting consensus metadata
I20260924 16:15:49.104548 20862 catalog_manager.cc:4841] TS d80304effe884cee9eafced5371101f4 (127.1.253.66:39115): tablet 6012852fde484986ae8b67b2ffc2e8cf (table test-table [id=c710e19b9cb646ea9a5831f2fce334d3]) successfully deleted
I20260924 16:15:49.105137 20861 catalog_manager.cc:4841] TS 24404c724f8e41f2969576ab6a4386a6 (127.1.253.67:38453): tablet 6012852fde484986ae8b67b2ffc2e8cf (table test-table [id=c710e19b9cb646ea9a5831f2fce334d3]) successfully deleted
W20260924 16:15:49.107817 20861 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:43289 (error 108)
I20260924 16:15:49.108635 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 21073
I20260924 16:15:49.108873 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:49.108563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:43289 (local address 127.0.0.1:43152)
0924 16:15:49.108698 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:49.108776 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:43289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 16:15:49.109076 20861 catalog_manager.cc:4584] TS e2b23919f3334ec284b485ee773fef89 (127.1.253.65:43289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6012852fde484986ae8b67b2ffc2e8cf: Network error: Client connection negotiation failed: client connection to 127.1.253.65:43289: connect: Connection refused (error 111)
I20260924 16:15:49.119213 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 21206
I20260924 16:15:49.125218 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 20844
2026-09-24T16:15:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1049 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:49 dist-test-slave-6wvm krb5kdc[21382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:49 dist-test-slave-6wvm krb5kdc[21382](info): set up 2 sockets
Sep 24 16:15:49 dist-test-slave-6wvm krb5kdc[21382](info): commencing operation
krb5kdc: starting...
W20260924 16:15:50.174114 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:15:50 dist-test-slave-6wvm krb5kdc[21382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:15:50.301559 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:41081
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:41081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:50.417991 21396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:50.418427 21396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:50.418558 21396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:50.422751 21396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:50.422820 21396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:50.422879 21396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:50.422917 21396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:50.422956 21396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:50.428591 21396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:41081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:41081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:50.429729 21396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:50.430898 21396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:50.439625 21404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:50.439646 21402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:50.439704 21401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:50.439900 21396 server_base.cc:1034] running on GCE node
I20260924 16:15:50.440634 21396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:50.441797 21396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:50.442991 21396 hybrid_clock.cc:648] HybridClock initialized: now 1790266550442978 us; error 32 us; skew 500 ppm
Sep 24 16:15:50 dist-test-slave-6wvm krb5kdc[21382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:50.446107 21396 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:15:50.447144 21396 webserver.cc:460] Webserver started at http://127.1.253.126:40629/ using document root <none> and password file <none>
I20260924 16:15:50.447788 21396 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:50.447851 21396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:50.448119 21396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:50.450208 21396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "c67469d0b48945299c1ca6a1efca9341"
format_stamp: "Formatted at 2026-09-24 16:15:50 on dist-test-slave-6wvm"
server_key: "83715e9bb3c080614453d6bb7d07cba4"
server_key_iv: "a29e6712e62deef91de1da154afbd23d"
server_key_version: "encryptionkey@0"
I20260924 16:15:50.450870 21396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "c67469d0b48945299c1ca6a1efca9341"
format_stamp: "Formatted at 2026-09-24 16:15:50 on dist-test-slave-6wvm"
server_key: "83715e9bb3c080614453d6bb7d07cba4"
server_key_iv: "a29e6712e62deef91de1da154afbd23d"
server_key_version: "encryptionkey@0"
I20260924 16:15:50.454838 21396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:15:50.457453 21411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:50.458828 21396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:15:50.459029 21396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "c67469d0b48945299c1ca6a1efca9341"
format_stamp: "Formatted at 2026-09-24 16:15:50 on dist-test-slave-6wvm"
server_key: "83715e9bb3c080614453d6bb7d07cba4"
server_key_iv: "a29e6712e62deef91de1da154afbd23d"
server_key_version: "encryptionkey@0"
I20260924 16:15:50.459156 21396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:50.475528 21396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:50.479230 21396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:50.479470 21396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:50.487094 21396 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:41081
I20260924 16:15:50.487143 21463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:41081 every 8 connection(s)
I20260924 16:15:50.488227 21396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:50.488492 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 21396
I20260924 16:15:50.488632 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:15:50.488996 2037 external_mini_cluster.cc:1428] Setting key a95b74b199eaaa4b6e79fc91572de18e
I20260924 16:15:50.491667 21464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:50.497568 21464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341: Bootstrap starting.
Sep 24 16:15:50 dist-test-slave-6wvm krb5kdc[21382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:15:50.500334 21464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:50.501152 21464 log.cc:824] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:50.503472 21464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341: No bootstrap required, opened a new log
I20260924 16:15:50.503547 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:50.490588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.1.253.126:41081)
0924 16:15:50.491060 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:50.491076 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:50.491118 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:15:50.491795 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:50.491822 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:50.492194 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:50.492548 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:50.493399 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:50.494468 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:50.495336 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:50.495829 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:50.496003 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:50.498627 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:50.498633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:50.498647 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:50.498675 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:50.500404 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:50.500902 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:50.500908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:50.500913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:50.501025 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:50.501323 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:50.501326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:50.501328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:50.501855 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:50.502065 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:50.502463 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 16:15:50.502802 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":173,"threads_started":1}
I20260924 16:15:50.507133 21464 raft_consensus.cc:348] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67469d0b48945299c1ca6a1efca9341" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41081 } }
I20260924 16:15:50.507400 21464 raft_consensus.cc:374] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:50.507510 21464 raft_consensus.cc:729] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67469d0b48945299c1ca6a1efca9341, State: Initialized, Role: FOLLOWER
I20260924 16:15:50.507974 21464 consensus_queue.cc:260] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67469d0b48945299c1ca6a1efca9341" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41081 } }
I20260924 16:15:50.508143 21464 raft_consensus.cc:388] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:50.508232 21464 raft_consensus.cc:482] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:50.508359 21464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:50.509358 21464 raft_consensus.cc:504] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67469d0b48945299c1ca6a1efca9341" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41081 } }
I20260924 16:15:50.509729 21464 leader_election.cc:302] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67469d0b48945299c1ca6a1efca9341; no voters:
I20260924 16:15:50.510106 21464 leader_election.cc:288] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:50.510205 21471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:50.510452 21471 raft_consensus.cc:686] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [term 1 LEADER]: Becoming Leader. State: Replica: c67469d0b48945299c1ca6a1efca9341, State: Running, Role: LEADER
I20260924 16:15:50.510764 21471 consensus_queue.cc:237] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67469d0b48945299c1ca6a1efca9341" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41081 } }
I20260924 16:15:50.511580 21464 sys_catalog.cc:564] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:50.512044 21473 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c67469d0b48945299c1ca6a1efca9341. Latest consensus state: current_term: 1 leader_uuid: "c67469d0b48945299c1ca6a1efca9341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67469d0b48945299c1ca6a1efca9341" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41081 } } }
I20260924 16:15:50.512272 21473 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:50.512550 21472 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c67469d0b48945299c1ca6a1efca9341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67469d0b48945299c1ca6a1efca9341" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41081 } } }
I20260924 16:15:50.512673 21472 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:50.512918 21476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:50.516463 21476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:50.522802 21476 catalog_manager.cc:1347] Generated new cluster ID: 59508fc658fa4bc880ad94c9e5fc1cb2
I20260924 16:15:50.522895 21476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:50.539472 21476 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:50.540812 21476 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:50.545846 21476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341: Generated new TSK 0
I20260924 16:15:50.546835 21476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.253.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
Entry for principal kudu/127.1.253.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab.
I20260924 16:15:50.589037 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41081
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:50.705331 21493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:50.705593 21493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:50.705636 21493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:50.709694 21493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:50.709750 21493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:50.709864 21493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:15:50.715543 21493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.65.keytab
--principal=kudu/127.1.253.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:50.716686 21493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:50.717801 21493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:50.728122 21501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:50.728206 21493 server_base.cc:1034] running on GCE node
W20260924 16:15:50.728188 21500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:50.728137 21503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:50.728897 21493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:50.729656 21493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:50.730873 21493 hybrid_clock.cc:648] HybridClock initialized: now 1790266550730833 us; error 54 us; skew 500 ppm
Sep 24 16:15:50 dist-test-slave-6wvm krb5kdc[21382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:50.733850 21493 init.cc:377] Logged in from keytab as kudu/127.1.253.65@KRBTEST.COM (short username kudu)
I20260924 16:15:50.734786 21493 webserver.cc:460] Webserver started at http://127.1.253.65:33973/ using document root <none> and password file <none>
I20260924 16:15:50.735347 21493 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:50.735406 21493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:50.735635 21493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:50.737766 21493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "4810fe3cff884bfa8c12d7be3fb0d599"
format_stamp: "Formatted at 2026-09-24 16:15:50 on dist-test-slave-6wvm"
server_key: "2ab3c908db57eef3ee17663f52fcdd47"
server_key_iv: "dca9ffe3692804bae33361fac7b0d8de"
server_key_version: "encryptionkey@0"
I20260924 16:15:50.738404 21493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "4810fe3cff884bfa8c12d7be3fb0d599"
format_stamp: "Formatted at 2026-09-24 16:15:50 on dist-test-slave-6wvm"
server_key: "2ab3c908db57eef3ee17663f52fcdd47"
server_key_iv: "dca9ffe3692804bae33361fac7b0d8de"
server_key_version: "encryptionkey@0"
I20260924 16:15:50.742408 21493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:15:50.744989 21510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:50.746366 21493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:50.746469 21493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "4810fe3cff884bfa8c12d7be3fb0d599"
format_stamp: "Formatted at 2026-09-24 16:15:50 on dist-test-slave-6wvm"
server_key: "2ab3c908db57eef3ee17663f52fcdd47"
server_key_iv: "dca9ffe3692804bae33361fac7b0d8de"
server_key_version: "encryptionkey@0"
I20260924 16:15:50.746608 21493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:50.763473 21493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:50.767021 21493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:50.767231 21493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:50.767854 21493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:50.768729 21493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:50.768776 21493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:50.768848 21493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:50.768880 21493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:50.777875 21493 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:46829
I20260924 16:15:50.777944 21624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:46829 every 8 connection(s)
I20260924 16:15:50.778903 21493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:50.786422 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 21493
I20260924 16:15:50.786581 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:15:50.786952 2037 external_mini_cluster.cc:1428] Setting key 0099e322f17dc4d9c43d4c1578d6f76d
Sep 24 16:15:50 dist-test-slave-6wvm krb5kdc[21382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.65@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:15:50.792702 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:50.780784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:37035 (local address 127.1.253.126:41081)
0924 16:15:50.780931 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:50.780935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:50.781808 (+ 873us) server_negotiation.cc:407] Connection header received
0924 16:15:50.782753 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:50.782755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:50.782806 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:50.782883 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:50.784056 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:50.784592 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:50.785694 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:50.785926 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:50.786009 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:50.788834 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:50.788837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:50.788839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:50.788862 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:50.790613 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:50.791152 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:50.791156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:50.791158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:50.791241 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:50.791588 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:50.791592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:50.791594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:50.791776 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:50.791886 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:50.792387 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 16:15:50.792529 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:15:50.793509 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:50.781180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:41081 (local address 127.1.253.65:37035)
0924 16:15:50.781645 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:50.781688 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:50.782532 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:50.783055 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:50.783062 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:50.783418 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:50.783912 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:50.783926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:50.784701 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:50.784703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:50.785563 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:50.785572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:50.786062 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:50.786064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:50.786326 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:50.786967 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:50.786993 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:50.788659 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:50.790753 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:50.790760 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:50.790775 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:50.791044 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:50.791335 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:50.791339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:50.791341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:50.791485 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:50.791885 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:50.791892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:50.792147 (+ 255us) client_negotiation.cc:769] Sending connection context
0924 16:15:50.792344 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 16:15:50.792669 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":107,"threads_started":1}
I20260924 16:15:50.795038 21625 heartbeater.cc:347] Connected to a master server at 127.1.253.126:41081
I20260924 16:15:50.795296 21625 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:50.795807 21625 heartbeater.cc:511] Master 127.1.253.126:41081 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.253.66@KRBTEST.COM; defaulting to no policy
I20260924 16:15:50.797490 21428 ts_manager.cc:194] Registered new tserver with Master: 4810fe3cff884bfa8c12d7be3fb0d599 (127.1.253.65:46829)
I20260924 16:15:50.798751 21428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.65@KRBTEST.COM'} at 127.1.253.65:37035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
Entry for principal kudu/127.1.253.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab.
I20260924 16:15:50.817649 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41081
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:50.932605 21630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:50.932920 21630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:50.933012 21630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:50.937156 21630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:50.937251 21630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:50.937347 21630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:15:50.942963 21630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.66.keytab
--principal=kudu/127.1.253.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:50.944170 21630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:50.945286 21630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:50.956081 21638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:50.956080 21636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:50.956104 21635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:50.956300 21630 server_base.cc:1034] running on GCE node
I20260924 16:15:50.957010 21630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:50.957729 21630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:50.958932 21630 hybrid_clock.cc:648] HybridClock initialized: now 1790266550958915 us; error 29 us; skew 500 ppm
Sep 24 16:15:50 dist-test-slave-6wvm krb5kdc[21382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:50.961941 21630 init.cc:377] Logged in from keytab as kudu/127.1.253.66@KRBTEST.COM (short username kudu)
I20260924 16:15:50.962983 21630 webserver.cc:460] Webserver started at http://127.1.253.66:42273/ using document root <none> and password file <none>
I20260924 16:15:50.963647 21630 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:50.963712 21630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:50.964021 21630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:50.966154 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "55efd06446e34835aa98ca16bf38763b"
format_stamp: "Formatted at 2026-09-24 16:15:50 on dist-test-slave-6wvm"
server_key: "abb5cbaa79da998a25f446a1d5ec4f52"
server_key_iv: "b148945c360ecc67776df42a783cf2f4"
server_key_version: "encryptionkey@0"
I20260924 16:15:50.966838 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "55efd06446e34835aa98ca16bf38763b"
format_stamp: "Formatted at 2026-09-24 16:15:50 on dist-test-slave-6wvm"
server_key: "abb5cbaa79da998a25f446a1d5ec4f52"
server_key_iv: "b148945c360ecc67776df42a783cf2f4"
server_key_version: "encryptionkey@0"
I20260924 16:15:50.970975 21630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:15:50.973567 21645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:50.974968 21630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:50.975127 21630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "55efd06446e34835aa98ca16bf38763b"
format_stamp: "Formatted at 2026-09-24 16:15:50 on dist-test-slave-6wvm"
server_key: "abb5cbaa79da998a25f446a1d5ec4f52"
server_key_iv: "b148945c360ecc67776df42a783cf2f4"
server_key_version: "encryptionkey@0"
I20260924 16:15:50.975250 21630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:50.996305 21630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:50.999428 21630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:50.999644 21630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:51.000277 21630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:51.001180 21630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:51.001253 21630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:51.001327 21630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:51.001380 21630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:51.010730 21630 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:43553
I20260924 16:15:51.010749 21760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:43553 every 8 connection(s)
I20260924 16:15:51.011655 21630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:51.014878 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 21630
I20260924 16:15:51.015017 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:15:51.015362 2037 external_mini_cluster.cc:1428] Setting key 819fe18053f0b3a00fde6c8bffc66578
Sep 24 16:15:51 dist-test-slave-6wvm krb5kdc[21382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266550, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.66@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.253.67@KRBTEST.COM; defaulting to no policy
I20260924 16:15:51.027706 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.013812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:38937 (local address 127.1.253.126:41081)
0924 16:15:51.013952 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:51.013956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:51.015162 (+ 1206us) server_negotiation.cc:407] Connection header received
0924 16:15:51.016217 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:51.016221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:51.016279 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:51.016356 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:51.017750 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.018735 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.019904 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.020334 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.020481 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.023580 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:51.023584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:51.023586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:51.023613 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:51.025185 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:51.025865 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:51.025868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:51.025869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:51.025943 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:51.026426 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:51.026428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:51.026430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:51.026616 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:51.026702 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:51.027468 (+ 766us) server_negotiation.cc:299] Negotiation successful
0924 16:15:51.027580 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:15:51.028774 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.014335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:41081 (local address 127.1.253.66:38937)
0924 16:15:51.014920 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:51.015003 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:51.016007 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:51.016537 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:51.016546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:51.017055 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:51.017577 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.017600 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.018868 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.018871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.019767 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.019776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.020495 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.020498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.020664 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.021333 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:51.021350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:51.023385 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:51.025321 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:51.025330 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:51.025351 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:51.025757 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:51.026048 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:51.026053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:51.026058 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:51.026323 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:51.026751 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:51.026759 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:51.027136 (+ 377us) client_negotiation.cc:769] Sending connection context
0924 16:15:51.027414 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 16:15:51.027780 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.67@KRBTEST.COM" created.
I20260924 16:15:51.030236 21761 heartbeater.cc:347] Connected to a master server at 127.1.253.126:41081
I20260924 16:15:51.030577 21761 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:51.031296 21761 heartbeater.cc:511] Master 127.1.253.126:41081 requested a full tablet report, sending...
I20260924 16:15:51.032724 21428 ts_manager.cc:194] Registered new tserver with Master: 55efd06446e34835aa98ca16bf38763b (127.1.253.66:43553)
I20260924 16:15:51.033602 21428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.66@KRBTEST.COM'} at 127.1.253.66:38937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
Entry for principal kudu/127.1.253.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab.
I20260924 16:15:51.046774 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41081
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:51.168924 21766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:51.169205 21766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:51.169252 21766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:51.173401 21766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:51.173466 21766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:51.173529 21766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:15:51.179030 21766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.67.keytab
--principal=kudu/127.1.253.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:41081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:51.180258 21766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:51.181428 21766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:51.191841 21774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:51.191831 21771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:51.191869 21772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:51.192267 21766 server_base.cc:1034] running on GCE node
I20260924 16:15:51.192695 21766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:51.193353 21766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:51.194533 21766 hybrid_clock.cc:648] HybridClock initialized: now 1790266551194519 us; error 31 us; skew 500 ppm
Sep 24 16:15:51 dist-test-slave-6wvm krb5kdc[21382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266551, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:51.197299 21766 init.cc:377] Logged in from keytab as kudu/127.1.253.67@KRBTEST.COM (short username kudu)
I20260924 16:15:51.198210 21766 webserver.cc:460] Webserver started at http://127.1.253.67:37239/ using document root <none> and password file <none>
I20260924 16:15:51.198825 21766 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:51.198885 21766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:51.199075 21766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:51.201226 21766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "f7a591ca6c4d4fd0a46202eecb9759cd"
format_stamp: "Formatted at 2026-09-24 16:15:51 on dist-test-slave-6wvm"
server_key: "4d6ebae36614b0e790d2d09891eef07d"
server_key_iv: "92e018d82f37f6536b8bc2e784451e40"
server_key_version: "encryptionkey@0"
I20260924 16:15:51.201793 21766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "f7a591ca6c4d4fd0a46202eecb9759cd"
format_stamp: "Formatted at 2026-09-24 16:15:51 on dist-test-slave-6wvm"
server_key: "4d6ebae36614b0e790d2d09891eef07d"
server_key_iv: "92e018d82f37f6536b8bc2e784451e40"
server_key_version: "encryptionkey@0"
I20260924 16:15:51.205675 21766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:15:51.208272 21781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:51.209611 21766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:15:51.209715 21766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "f7a591ca6c4d4fd0a46202eecb9759cd"
format_stamp: "Formatted at 2026-09-24 16:15:51 on dist-test-slave-6wvm"
server_key: "4d6ebae36614b0e790d2d09891eef07d"
server_key_iv: "92e018d82f37f6536b8bc2e784451e40"
server_key_version: "encryptionkey@0"
I20260924 16:15:51.209808 21766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:51.256353 21766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:51.259877 21766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:51.260051 21766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:51.260768 21766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:51.261677 21766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:51.261724 21766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:51.261801 21766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:51.261842 21766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:51.271078 21766 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:38535
I20260924 16:15:51.271111 21895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:38535 every 8 connection(s)
I20260924 16:15:51.271992 21766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:51.274906 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 21766
I20260924 16:15:51.275067 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:15:51.275420 2037 external_mini_cluster.cc:1428] Setting key 674490c94c3e9acdbaf8fab2bbc4da57
Sep 24 16:15:51 dist-test-slave-6wvm krb5kdc[21382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266551, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.67@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:15:51.285786 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.273858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:45311 (local address 127.1.253.126:41081)
0924 16:15:51.274048 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:51.274052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:51.275023 (+ 971us) server_negotiation.cc:407] Connection header received
0924 16:15:51.276217 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:51.276220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:51.276281 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:51.276363 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:51.277869 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.278535 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.279754 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.279968 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.280088 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.282546 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:51.282548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:51.282550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:51.282572 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:51.283843 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:51.284383 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:51.284386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:51.284387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:51.284462 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:51.284839 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:51.284841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:51.284842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:51.285008 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:51.285082 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:51.285567 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 16:15:51.285663 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 16:15:51.286780 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.274278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:41081 (local address 127.1.253.67:45311)
0924 16:15:51.274801 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:51.274876 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:51.275967 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:51.276593 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:51.276602 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:51.277105 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:51.277717 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.277739 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.278653 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.278655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.279649 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.279658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.280122 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.280125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.280262 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.280888 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:51.280932 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:51.282393 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:51.283992 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:51.283998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:51.284016 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:51.284280 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:51.284569 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:51.284573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:51.284574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:51.284743 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:51.285101 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:51.285106 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:51.285346 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 16:15:51.285523 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 16:15:51.285783 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
I20260924 16:15:51.288028 21896 heartbeater.cc:347] Connected to a master server at 127.1.253.126:41081
I20260924 16:15:51.288280 21896 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:51.288808 21896 heartbeater.cc:511] Master 127.1.253.126:41081 requested a full tablet report, sending...
I20260924 16:15:51.289844 21428 ts_manager.cc:194] Registered new tserver with Master: f7a591ca6c4d4fd0a46202eecb9759cd (127.1.253.67:38535)
I20260924 16:15:51.290464 21428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.253.67@KRBTEST.COM'} at 127.1.253.67:45311
I20260924 16:15:51.301193 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:15:51.310820 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.303214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.1.253.126:41081)
0924 16:15:51.303317 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:51.303320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:51.303581 (+ 261us) server_negotiation.cc:407] Connection header received
0924 16:15:51.303735 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:51.303738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:51.303785 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:51.303873 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:51.304578 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.305098 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.305952 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.306186 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.306317 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.307761 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:51.307765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:51.307767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:51.307795 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:51.309090 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:51.309521 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:51.309525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:51.309527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:51.309615 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:51.309891 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:51.309896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:51.309901 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:51.310128 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:51.310290 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:51.310539 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 16:15:51.310659 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 16:15:51.313948 21428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:15:51.317114 21428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:15:51.331723 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.325550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:46829 (local address 127.0.0.1:33990)
0924 16:15:51.325958 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:51.326003 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:51.326121 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:51.327638 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:51.327642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:51.327662 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:51.327751 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.327756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.328845 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.328851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.330317 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.330328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.331161 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.331165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.331277 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.331313 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:15:51.331451 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 16:15:51.331579 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":58,"thread_start_us":94,"threads_started":1}
I20260924 16:15:51.331933 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.325645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:38535 (local address 127.0.0.1:42636)
0924 16:15:51.326668 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:51.326684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:51.326774 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:51.327429 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:51.327440 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:51.327458 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:51.327563 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.327567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.329189 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.329192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.330473 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.330482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.331494 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.331497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.331567 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.331584 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 16:15:51.331629 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 16:15:51.331729 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":64,"thread_start_us":103,"threads_started":1}
I20260924 16:15:51.332201 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.326532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.1.253.66:43553)
0924 16:15:51.327168 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:51.327186 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:51.327321 (+ 135us) server_negotiation.cc:407] Connection header received
0924 16:15:51.327429 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:51.327433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:51.327592 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:51.327718 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:51.328126 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.329223 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.330338 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.331114 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.331227 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.331360 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:51.331940 (+ 580us) server_negotiation.cc:299] Negotiation successful
0924 16:15:51.332052 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":103,"threads_started":1}
I20260924 16:15:51.332230 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.326881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.1.253.65:46829)
0924 16:15:51.327168 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:51.327177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:51.327203 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:15:51.327303 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:51.327309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:51.327498 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:51.327664 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:51.327869 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.328686 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.330460 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.331034 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.331161 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.331312 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:51.331489 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 16:15:51.331578 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260924 16:15:51.332511 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.326367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:43553 (local address 127.0.0.1:45526)
0924 16:15:51.327212 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:51.327225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:51.327328 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:51.327877 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:51.327880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:51.327899 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:51.327999 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.328005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.329357 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.329360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.330173 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.330182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.331441 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.331448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.331542 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.331567 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:15:51.331634 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 16:15:51.332395 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
I20260924 16:15:51.332613 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.326365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.1.253.67:38535)
0924 16:15:51.326881 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:51.326894 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:51.326919 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:15:51.327003 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:51.327009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:51.327242 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:51.327737 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:51.327897 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.329047 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.330584 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.331369 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.332175 (+ 806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.332325 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:51.332376 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 16:15:51.332464 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":152,"threads_started":1}
I20260924 16:15:51.335599 21830 tablet_service.cc:1467] Processing CreateTablet for tablet da17bea8fd664ac497343b77956ab0f6 (DEFAULT_TABLE table=test-table [id=57f9c5d9b9a449418d6756f8a68e717d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:51.335599 21694 tablet_service.cc:1467] Processing CreateTablet for tablet da17bea8fd664ac497343b77956ab0f6 (DEFAULT_TABLE table=test-table [id=57f9c5d9b9a449418d6756f8a68e717d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:51.336786 21694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da17bea8fd664ac497343b77956ab0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:51.336796 21830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da17bea8fd664ac497343b77956ab0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:51.337029 21559 tablet_service.cc:1467] Processing CreateTablet for tablet da17bea8fd664ac497343b77956ab0f6 (DEFAULT_TABLE table=test-table [id=57f9c5d9b9a449418d6756f8a68e717d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:51.338001 21559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da17bea8fd664ac497343b77956ab0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:51.344214 21914 tablet_bootstrap.cc:492] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599: Bootstrap starting.
I20260924 16:15:51.345077 21916 tablet_bootstrap.cc:492] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b: Bootstrap starting.
I20260924 16:15:51.345105 21915 tablet_bootstrap.cc:492] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd: Bootstrap starting.
I20260924 16:15:51.347186 21915 tablet_bootstrap.cc:654] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:51.347186 21914 tablet_bootstrap.cc:654] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:51.347543 21916 tablet_bootstrap.cc:654] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:51.348171 21914 log.cc:824] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:51.348208 21916 log.cc:824] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:51.348335 21915 log.cc:824] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:51.350334 21915 tablet_bootstrap.cc:492] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd: No bootstrap required, opened a new log
I20260924 16:15:51.350520 21914 tablet_bootstrap.cc:492] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599: No bootstrap required, opened a new log
I20260924 16:15:51.350603 21916 tablet_bootstrap.cc:492] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b: No bootstrap required, opened a new log
I20260924 16:15:51.350548 21915 ts_tablet_manager.cc:1397] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260924 16:15:51.350750 21914 ts_tablet_manager.cc:1397] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 16:15:51.350759 21916 ts_tablet_manager.cc:1397] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:15:51.354580 21914 raft_consensus.cc:348] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4810fe3cff884bfa8c12d7be3fb0d599" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 46829 } } peers { permanent_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38535 } } peers { permanent_uuid: "55efd06446e34835aa98ca16bf38763b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43553 } }
I20260924 16:15:51.354588 21916 raft_consensus.cc:348] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4810fe3cff884bfa8c12d7be3fb0d599" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 46829 } } peers { permanent_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38535 } } peers { permanent_uuid: "55efd06446e34835aa98ca16bf38763b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43553 } }
I20260924 16:15:51.354588 21915 raft_consensus.cc:348] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4810fe3cff884bfa8c12d7be3fb0d599" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 46829 } } peers { permanent_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38535 } } peers { permanent_uuid: "55efd06446e34835aa98ca16bf38763b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43553 } }
I20260924 16:15:51.354825 21914 raft_consensus.cc:374] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:51.354840 21915 raft_consensus.cc:374] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:51.354840 21916 raft_consensus.cc:374] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:51.354887 21914 raft_consensus.cc:729] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4810fe3cff884bfa8c12d7be3fb0d599, State: Initialized, Role: FOLLOWER
I20260924 16:15:51.354892 21915 raft_consensus.cc:729] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a591ca6c4d4fd0a46202eecb9759cd, State: Initialized, Role: FOLLOWER
I20260924 16:15:51.354899 21916 raft_consensus.cc:729] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55efd06446e34835aa98ca16bf38763b, State: Initialized, Role: FOLLOWER
I20260924 16:15:51.355322 21914 consensus_queue.cc:260] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4810fe3cff884bfa8c12d7be3fb0d599" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 46829 } } peers { permanent_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38535 } } peers { permanent_uuid: "55efd06446e34835aa98ca16bf38763b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43553 } }
I20260924 16:15:51.355329 21916 consensus_queue.cc:260] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4810fe3cff884bfa8c12d7be3fb0d599" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 46829 } } peers { permanent_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38535 } } peers { permanent_uuid: "55efd06446e34835aa98ca16bf38763b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43553 } }
I20260924 16:15:51.355329 21915 consensus_queue.cc:260] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4810fe3cff884bfa8c12d7be3fb0d599" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 46829 } } peers { permanent_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38535 } } peers { permanent_uuid: "55efd06446e34835aa98ca16bf38763b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43553 } }
I20260924 16:15:51.356361 21916 ts_tablet_manager.cc:1428] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:15:51.356361 21915 ts_tablet_manager.cc:1428] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260924 16:15:51.356420 21761 heartbeater.cc:503] Master 127.1.253.126:41081 was elected leader, sending a full tablet report...
I20260924 16:15:51.356431 21896 heartbeater.cc:503] Master 127.1.253.126:41081 was elected leader, sending a full tablet report...
I20260924 16:15:51.357466 21914 ts_tablet_manager.cc:1428] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 16:15:51.357586 21625 heartbeater.cc:503] Master 127.1.253.126:41081 was elected leader, sending a full tablet report...
I20260924 16:15:51.418251 21921 raft_consensus.cc:482] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:51.418521 21921 raft_consensus.cc:504] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4810fe3cff884bfa8c12d7be3fb0d599" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 46829 } } peers { permanent_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38535 } } peers { permanent_uuid: "55efd06446e34835aa98ca16bf38763b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43553 } }
I20260924 16:15:51.419785 21921 leader_election.cc:288] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4810fe3cff884bfa8c12d7be3fb0d599 (127.1.253.65:46829), f7a591ca6c4d4fd0a46202eecb9759cd (127.1.253.67:38535)
I20260924 16:15:51.424059 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.420196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:34771 (local address 127.1.253.65:46829)
0924 16:15:51.420320 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:51.420325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:51.420342 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:15:51.420421 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:51.420426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:51.420493 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:51.420582 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:51.420981 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.421656 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.422922 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.423503 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.423602 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.423645 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:51.423871 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 16:15:51.423930 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:15:51.424062 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.420053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:46829 (local address 127.1.253.66:34771)
0924 16:15:51.420203 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:51.420222 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:51.420321 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:51.420678 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:51.420682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:51.420704 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:51.420831 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.420837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.421773 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.421776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.422788 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.422798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.423618 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.423622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.423740 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.423767 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:15:51.423847 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 16:15:51.423922 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 16:15:51.424676 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.420073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:38535 (local address 127.1.253.66:45213)
0924 16:15:51.420784 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:51.420797 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:51.420918 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:51.421224 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:51.421227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:51.421245 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:51.421373 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.421389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.422322 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.422325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.423261 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:51.423272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.424267 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.424273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:51.424389 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.424409 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 16:15:51.424471 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 16:15:51.424518 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":263,"thread_start_us":138,"threads_started":1}
I20260924 16:15:51.424695 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.420184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:45213 (local address 127.1.253.67:38535)
0924 16:15:51.420321 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:51.420326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:51.420889 (+ 563us) server_negotiation.cc:407] Connection header received
0924 16:15:51.421053 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:51.421058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:51.421111 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:51.421212 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:51.421492 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.422174 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.423389 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.423975 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.424093 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.424147 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:51.424521 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 16:15:51.424607 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:15:51.424916 21579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da17bea8fd664ac497343b77956ab0f6" candidate_uuid: "55efd06446e34835aa98ca16bf38763b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4810fe3cff884bfa8c12d7be3fb0d599" is_pre_election: true
I20260924 16:15:51.425294 21579 raft_consensus.cc:2454] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55efd06446e34835aa98ca16bf38763b in term 0.
I20260924 16:15:51.425432 21850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da17bea8fd664ac497343b77956ab0f6" candidate_uuid: "55efd06446e34835aa98ca16bf38763b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd" is_pre_election: true
I20260924 16:15:51.425716 21850 raft_consensus.cc:2454] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55efd06446e34835aa98ca16bf38763b in term 0.
I20260924 16:15:51.425884 21648 leader_election.cc:302] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4810fe3cff884bfa8c12d7be3fb0d599, 55efd06446e34835aa98ca16bf38763b; no voters:
I20260924 16:15:51.426206 21921 raft_consensus.cc:2781] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:15:51.426307 21921 raft_consensus.cc:482] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:15:51.426376 21921 raft_consensus.cc:3037] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:51.427415 21921 raft_consensus.cc:504] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4810fe3cff884bfa8c12d7be3fb0d599" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 46829 } } peers { permanent_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38535 } } peers { permanent_uuid: "55efd06446e34835aa98ca16bf38763b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43553 } }
I20260924 16:15:51.427798 21921 leader_election.cc:288] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [CANDIDATE]: Term 1 election: Requested vote from peers 4810fe3cff884bfa8c12d7be3fb0d599 (127.1.253.65:46829), f7a591ca6c4d4fd0a46202eecb9759cd (127.1.253.67:38535)
I20260924 16:15:51.428148 21579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da17bea8fd664ac497343b77956ab0f6" candidate_uuid: "55efd06446e34835aa98ca16bf38763b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4810fe3cff884bfa8c12d7be3fb0d599"
I20260924 16:15:51.428148 21850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da17bea8fd664ac497343b77956ab0f6" candidate_uuid: "55efd06446e34835aa98ca16bf38763b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd"
I20260924 16:15:51.428282 21579 raft_consensus.cc:3037] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:51.428282 21850 raft_consensus.cc:3037] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:51.429772 21850 raft_consensus.cc:2454] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55efd06446e34835aa98ca16bf38763b in term 1.
I20260924 16:15:51.429772 21579 raft_consensus.cc:2454] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55efd06446e34835aa98ca16bf38763b in term 1.
I20260924 16:15:51.430125 21649 leader_election.cc:302] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55efd06446e34835aa98ca16bf38763b, f7a591ca6c4d4fd0a46202eecb9759cd; no voters:
I20260924 16:15:51.430337 21921 raft_consensus.cc:2781] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:51.430544 21921 raft_consensus.cc:686] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 1 LEADER]: Becoming Leader. State: Replica: 55efd06446e34835aa98ca16bf38763b, State: Running, Role: LEADER
I20260924 16:15:51.430874 21921 consensus_queue.cc:237] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4810fe3cff884bfa8c12d7be3fb0d599" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 46829 } } peers { permanent_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38535 } } peers { permanent_uuid: "55efd06446e34835aa98ca16bf38763b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43553 } }
I20260924 16:15:51.434414 21428 catalog_manager.cc:5495] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b reported cstate change: term changed from 0 to 1, leader changed from <none> to 55efd06446e34835aa98ca16bf38763b (127.1.253.66). New cstate: current_term: 1 leader_uuid: "55efd06446e34835aa98ca16bf38763b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4810fe3cff884bfa8c12d7be3fb0d599" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 46829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55efd06446e34835aa98ca16bf38763b" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 43553 } health_report { overall_health: HEALTHY } } }
I20260924 16:15:51.454351 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.450503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.1.253.66:43553)
0924 16:15:51.450652 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:51.450657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:51.450673 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:15:51.450880 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:51.450884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:51.450965 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:51.451095 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:15:51.451380 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.451916 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.452812 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.453038 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.453151 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.453482 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:15:51.453918 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:15:51.454057 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:51.454101 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 16:15:51.454160 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:15:51.460598 21579 raft_consensus.cc:1261] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599 [term 1 FOLLOWER]: Refusing update from remote peer 55efd06446e34835aa98ca16bf38763b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:51.460604 21850 raft_consensus.cc:1261] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd [term 1 FOLLOWER]: Refusing update from remote peer 55efd06446e34835aa98ca16bf38763b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:51.461454 21921 consensus_queue.cc:1035] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4810fe3cff884bfa8c12d7be3fb0d599" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 46829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:51.461668 21924 consensus_queue.cc:1035] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7a591ca6c4d4fd0a46202eecb9759cd" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 38535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:51.469337 21927 mvcc.cc:205] Tried to move back new op lower bound from 7332931794774523904 to 7332931794663407616. Current Snapshot: MvccSnapshot[applied={T|T < 7332931794774523904}]
I20260924 16:15:51.470556 21929 mvcc.cc:205] Tried to move back new op lower bound from 7332931794774523904 to 7332931794663407616. Current Snapshot: MvccSnapshot[applied={T|T < 7332931794774523904}]
I20260924 16:15:51.480325 21427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:51.480530 21427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:51.483176 21427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c67469d0b48945299c1ca6a1efca9341: Sending DeleteTablet for 3 replicas of tablet da17bea8fd664ac497343b77956ab0f6
I20260924 16:15:51.484005 21559 tablet_service.cc:1514] Processing DeleteTablet for tablet da17bea8fd664ac497343b77956ab0f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:51 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:33990
I20260924 16:15:51.484035 21694 tablet_service.cc:1514] Processing DeleteTablet for tablet da17bea8fd664ac497343b77956ab0f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:51 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:45526
I20260924 16:15:51.484527 21937 tablet_replica.cc:330] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b: stopping tablet replica
I20260924 16:15:51.484543 21830 tablet_service.cc:1514] Processing DeleteTablet for tablet da17bea8fd664ac497343b77956ab0f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:51 UTC) from {username='kudu', principal='kudu/127.1.253.126@KRBTEST.COM'} at 127.0.0.1:42636
I20260924 16:15:51.484856 21937 raft_consensus.cc:2229] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:15:51.484942 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 21493
I20260924 16:15:51.485162 21938 tablet_replica.cc:330] T da17bea8fd664ac497343b77956ab0f6 P 4810fe3cff884bfa8c12d7be3fb0d599: stopping tablet replica
I20260924 16:15:51.485291 21937 raft_consensus.cc:2258] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:51.487069 21937 ts_tablet_manager.cc:1905] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:51.490697 21937 ts_tablet_manager.cc:1918] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:51.490799 21937 log.cc:1196] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/da17bea8fd664ac497343b77956ab0f6
I20260924 16:15:51.491132 21937 ts_tablet_manager.cc:1939] T da17bea8fd664ac497343b77956ab0f6 P 55efd06446e34835aa98ca16bf38763b: Deleting consensus metadata
I20260924 16:15:51.492463 21415 catalog_manager.cc:4841] TS 55efd06446e34835aa98ca16bf38763b (127.1.253.66:43553): tablet da17bea8fd664ac497343b77956ab0f6 (table test-table [id=57f9c5d9b9a449418d6756f8a68e717d]) successfully deleted
I20260924 16:15:51.494501 21939 tablet_replica.cc:330] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd: stopping tablet replica
I20260924 16:15:51.494912 21939 raft_consensus.cc:2229] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:51.495270 21939 raft_consensus.cc:2258] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 16:15:51.495972 21414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:46829 (error 108)
I20260924 16:15:51.496830 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 21630
I20260924 16:15:51.496833 21939 ts_tablet_manager.cc:1905] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:51.499761 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.499460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:46829 (local address 127.0.0.1:33994)
0924 16:15:51.499616 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:51.499686 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:46829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 16:15:51.499928 21414 catalog_manager.cc:4584] TS 4810fe3cff884bfa8c12d7be3fb0d599 (127.1.253.65:46829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da17bea8fd664ac497343b77956ab0f6: Network error: Client connection negotiation failed: client connection to 127.1.253.65:46829: connect: Connection refused (error 111)
I20260924 16:15:51.501461 21939 ts_tablet_manager.cc:1918] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:51.501564 21939 log.cc:1196] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/da17bea8fd664ac497343b77956ab0f6
I20260924 16:15:51.501889 21939 ts_tablet_manager.cc:1939] T da17bea8fd664ac497343b77956ab0f6 P f7a591ca6c4d4fd0a46202eecb9759cd: Deleting consensus metadata
I20260924 16:15:51.503278 21415 catalog_manager.cc:4841] TS f7a591ca6c4d4fd0a46202eecb9759cd (127.1.253.67:38535): tablet da17bea8fd664ac497343b77956ab0f6 (table test-table [id=57f9c5d9b9a449418d6756f8a68e717d]) successfully deleted
I20260924 16:15:51.505897 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 21766
I20260924 16:15:51.512496 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 21396
2026-09-24T16:15:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2389 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T16:15:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:51Z Disabled control of system clock
I20260924 16:15:51.544826 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:43575
--webserver_interface=127.1.253.126
--webserver_port=0
--builtin_ntp_servers=127.1.253.84:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:15:51.658658 21944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:51.658914 21944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:51.659003 21944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:51.663153 21944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:51.663214 21944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:51.663230 21944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:51.663281 21944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:51.663302 21944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:51.668112 21944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:43575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:43575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:51.669191 21944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:51.670594 21944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:51.678766 21950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:51.678828 21949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:51.678785 21952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:51.678975 21944 server_base.cc:1034] running on GCE node
I20260924 16:15:51.679690 21944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:51.680838 21944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:51.682003 21944 hybrid_clock.cc:648] HybridClock initialized: now 1790266551681985 us; error 29 us; skew 500 ppm
I20260924 16:15:51.683967 21944 webserver.cc:460] Webserver started at http://127.1.253.126:37885/ using document root <none> and password file <none>
I20260924 16:15:51.684571 21944 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:51.684634 21944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:51.684880 21944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:51.687107 21944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "460a5de8d34148a8994ea678f16701cf"
format_stamp: "Formatted at 2026-09-24 16:15:51 on dist-test-slave-6wvm"
server_key: "66dcb8710fc5de42bdaf4f7190a0a527"
server_key_iv: "9d41c93a7eb76bbc7d70421c327e384a"
server_key_version: "encryptionkey@0"
I20260924 16:15:51.687697 21944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "460a5de8d34148a8994ea678f16701cf"
format_stamp: "Formatted at 2026-09-24 16:15:51 on dist-test-slave-6wvm"
server_key: "66dcb8710fc5de42bdaf4f7190a0a527"
server_key_iv: "9d41c93a7eb76bbc7d70421c327e384a"
server_key_version: "encryptionkey@0"
I20260924 16:15:51.691643 21944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:15:51.694108 21958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:51.695446 21944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:15:51.695592 21944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "460a5de8d34148a8994ea678f16701cf"
format_stamp: "Formatted at 2026-09-24 16:15:51 on dist-test-slave-6wvm"
server_key: "66dcb8710fc5de42bdaf4f7190a0a527"
server_key_iv: "9d41c93a7eb76bbc7d70421c327e384a"
server_key_version: "encryptionkey@0"
I20260924 16:15:51.695710 21944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:51.731645 21944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:51.732443 21944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:51.732681 21944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:51.739864 21944 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:43575
I20260924 16:15:51.739874 22011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:43575 every 8 connection(s)
I20260924 16:15:51.740875 21944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:51.741093 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 21944
I20260924 16:15:51.741242 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:15:51.741577 2037 external_mini_cluster.cc:1428] Setting key 4cf6925b25eff4689785655bba8a8f0d
I20260924 16:15:51.744521 22012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:51.750890 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.743061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.1.253.126:43575)
0924 16:15:51.743702 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:51.743722 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:51.743758 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:15:51.744408 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:51.744425 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:51.744873 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:51.745164 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:51.745499 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.746509 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.747694 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:51.748157 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:51.748270 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:51.749187 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:51.749192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:51.749204 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:51.749573 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:51.749627 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:51.749647 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:51.749723 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:51.749880 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 16:15:51.750163 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":137,"threads_started":1}
I20260924 16:15:51.751152 22012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf: Bootstrap starting.
I20260924 16:15:51.753417 22012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:51.754148 22012 log.cc:824] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:51.756075 22012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf: No bootstrap required, opened a new log
I20260924 16:15:51.759647 22012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460a5de8d34148a8994ea678f16701cf" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43575 } }
I20260924 16:15:51.759881 22012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:51.759960 22012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460a5de8d34148a8994ea678f16701cf, State: Initialized, Role: FOLLOWER
I20260924 16:15:51.760375 22012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460a5de8d34148a8994ea678f16701cf" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43575 } }
I20260924 16:15:51.760529 22012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:51.760630 22012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:51.760751 22012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:51.761600 22012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460a5de8d34148a8994ea678f16701cf" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43575 } }
I20260924 16:15:51.761935 22012 leader_election.cc:302] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 460a5de8d34148a8994ea678f16701cf; no voters:
I20260924 16:15:51.762274 22012 leader_election.cc:288] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:51.762391 22017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:51.762639 22017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [term 1 LEADER]: Becoming Leader. State: Replica: 460a5de8d34148a8994ea678f16701cf, State: Running, Role: LEADER
I20260924 16:15:51.762941 22017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460a5de8d34148a8994ea678f16701cf" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43575 } }
I20260924 16:15:51.763567 22012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:51.764659 22019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 460a5de8d34148a8994ea678f16701cf. Latest consensus state: current_term: 1 leader_uuid: "460a5de8d34148a8994ea678f16701cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460a5de8d34148a8994ea678f16701cf" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43575 } } }
I20260924 16:15:51.764708 22018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "460a5de8d34148a8994ea678f16701cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460a5de8d34148a8994ea678f16701cf" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43575 } } }
I20260924 16:15:51.764788 22019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:51.764818 22018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:51.765152 22027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:51.768484 22027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:51.774173 22027 catalog_manager.cc:1347] Generated new cluster ID: 433f59fbb1274eeab302841b5872b170
I20260924 16:15:51.774262 22027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:51.791882 22027 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:51.792951 22027 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:51.804692 22027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf: Generated new TSK 0
I20260924 16:15:51.805424 22027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:15:51.815603 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.65:0
--local_ip_for_outbound_sockets=127.1.253.65
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43575
--builtin_ntp_servers=127.1.253.84:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:15:51.940908 22037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:51.941179 22037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:51.941217 22037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:51.945353 22037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:51.945405 22037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:51.945482 22037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.65
I20260924 16:15:51.950177 22037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.253.65
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.65
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:51.951321 22037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:51.952782 22037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:15:51.962474 22037 server_base.cc:1034] running on GCE node
W20260924 16:15:51.962435 22042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:51.962435 22045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:51.962435 22043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:51.963227 22037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:51.963902 22037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:51.965132 22037 hybrid_clock.cc:648] HybridClock initialized: now 1790266551965075 us; error 60 us; skew 500 ppm
I20260924 16:15:51.967257 22037 webserver.cc:460] Webserver started at http://127.1.253.65:42771/ using document root <none> and password file <none>
I20260924 16:15:51.967875 22037 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:51.967972 22037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:51.968211 22037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:51.970383 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/data/instance:
uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4"
format_stamp: "Formatted at 2026-09-24 16:15:51 on dist-test-slave-6wvm"
server_key: "8e2409660cfb2e736f6dc0936cbd7716"
server_key_iv: "14d65055d7bc9b21b2b853f6632efabf"
server_key_version: "encryptionkey@0"
I20260924 16:15:51.970970 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance:
uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4"
format_stamp: "Formatted at 2026-09-24 16:15:51 on dist-test-slave-6wvm"
server_key: "8e2409660cfb2e736f6dc0936cbd7716"
server_key_iv: "14d65055d7bc9b21b2b853f6632efabf"
server_key_version: "encryptionkey@0"
I20260924 16:15:51.974682 22037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:15:51.977059 22051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:51.978222 22037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:15:51.978412 22037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/wal
uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4"
format_stamp: "Formatted at 2026-09-24 16:15:51 on dist-test-slave-6wvm"
server_key: "8e2409660cfb2e736f6dc0936cbd7716"
server_key_iv: "14d65055d7bc9b21b2b853f6632efabf"
server_key_version: "encryptionkey@0"
I20260924 16:15:51.978533 22037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:51.995851 22037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:51.996546 22037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:51.996752 22037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:51.997352 22037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:51.998385 22037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:51.998451 22037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:51.998531 22037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:51.998574 22037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:52.007774 22037 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.65:36707
I20260924 16:15:52.007798 22167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.65:36707 every 8 connection(s)
I20260924 16:15:52.008695 22037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:52.013439 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 22037
I20260924 16:15:52.013581 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-0/wal/instance
I20260924 16:15:52.013897 2037 external_mini_cluster.cc:1428] Setting key a40e234c26d104594547eab946975d3c
I20260924 16:15:52.016139 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.66:0
--local_ip_for_outbound_sockets=127.1.253.66
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43575
--builtin_ntp_servers=127.1.253.84:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:15:52.019497 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.010599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.65:56189 (local address 127.1.253.126:43575)
0924 16:15:52.010740 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:52.010744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:52.011764 (+ 1020us) server_negotiation.cc:407] Connection header received
0924 16:15:52.013094 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:52.013097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:52.013152 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:52.013227 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:52.014929 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.015682 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.017442 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.017658 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.017771 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.018945 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:52.018949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:52.018952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:52.018972 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:52.019001 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:52.019009 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:52.019090 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:52.019256 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 16:15:52.019346 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":49}
I20260924 16:15:52.020345 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.010956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:43575 (local address 127.1.253.65:56189)
0924 16:15:52.011616 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:52.011656 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:52.012902 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:52.013917 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:52.013928 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:52.014047 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:52.014776 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.014794 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.015819 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.015823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.017300 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.017316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.017827 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.017832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.017999 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.018676 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:15:52.018677 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:52.018718 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:15:52.018784 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:52.019092 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:52.019104 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:52.019113 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:15:52.019252 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 16:15:52.019558 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260924 16:15:52.021697 22168 heartbeater.cc:347] Connected to a master server at 127.1.253.126:43575
I20260924 16:15:52.021955 22168 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:52.022560 22168 heartbeater.cc:511] Master 127.1.253.126:43575 requested a full tablet report, sending...
I20260924 16:15:52.024113 21976 ts_manager.cc:194] Registered new tserver with Master: 3d1fdc1ca55d49b6a55a98d5bf2d1fb4 (127.1.253.65:36707)
I20260924 16:15:52.025259 21976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.253.65:56189
W20260924 16:15:52.134117 22171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:52.134414 22171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:52.134476 22171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:52.138445 22171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:52.138490 22171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:52.138542 22171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.66
I20260924 16:15:52.143397 22171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.253.66
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.66
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:52.144490 22171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:52.145967 22171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:52.155586 22176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:52.155604 22177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:52.155586 22179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:52.155810 22171 server_base.cc:1034] running on GCE node
I20260924 16:15:52.156334 22171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:52.157001 22171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:52.158215 22171 hybrid_clock.cc:648] HybridClock initialized: now 1790266552158199 us; error 32 us; skew 500 ppm
I20260924 16:15:52.160313 22171 webserver.cc:460] Webserver started at http://127.1.253.66:44029/ using document root <none> and password file <none>
I20260924 16:15:52.161000 22171 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:52.161104 22171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:52.161363 22171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:52.163779 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/data/instance:
uuid: "5e809f33654647328263c56ce2790bf5"
format_stamp: "Formatted at 2026-09-24 16:15:52 on dist-test-slave-6wvm"
server_key: "d8e216632bb6d0211b244873311af5df"
server_key_iv: "90eac9e4f75c604678c20cc642bf6a57"
server_key_version: "encryptionkey@0"
I20260924 16:15:52.164455 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance:
uuid: "5e809f33654647328263c56ce2790bf5"
format_stamp: "Formatted at 2026-09-24 16:15:52 on dist-test-slave-6wvm"
server_key: "d8e216632bb6d0211b244873311af5df"
server_key_iv: "90eac9e4f75c604678c20cc642bf6a57"
server_key_version: "encryptionkey@0"
I20260924 16:15:52.168702 22171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 16:15:52.171301 22186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:52.172561 22171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:15:52.172713 22171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/wal
uuid: "5e809f33654647328263c56ce2790bf5"
format_stamp: "Formatted at 2026-09-24 16:15:52 on dist-test-slave-6wvm"
server_key: "d8e216632bb6d0211b244873311af5df"
server_key_iv: "90eac9e4f75c604678c20cc642bf6a57"
server_key_version: "encryptionkey@0"
I20260924 16:15:52.172837 22171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:52.195012 22171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:52.195778 22171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:52.196014 22171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:52.196673 22171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:52.197719 22171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:52.197798 22171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:52.197877 22171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:52.197928 22171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:52.207571 22171 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.66:37875
I20260924 16:15:52.207602 22301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.66:37875 every 8 connection(s)
I20260924 16:15:52.208544 22171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:52.213538 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 22171
I20260924 16:15:52.213702 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/wal/instance
I20260924 16:15:52.214007 2037 external_mini_cluster.cc:1428] Setting key f2c83c49019cfa0b310e62591b30dff5
I20260924 16:15:52.216068 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.253.67:0
--local_ip_for_outbound_sockets=127.1.253.67
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43575
--builtin_ntp_servers=127.1.253.84:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:15:52.218710 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.210511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:38077 (local address 127.1.253.126:43575)
0924 16:15:52.210673 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:52.210677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:52.211755 (+ 1078us) server_negotiation.cc:407] Connection header received
0924 16:15:52.213160 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:52.213162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:52.213213 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:52.213323 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:52.214461 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.215085 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.216451 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.216856 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.216981 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.218189 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:52.218192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:52.218194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:52.218206 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:52.218224 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:52.218232 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:52.218348 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:52.218535 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 16:15:52.218598 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:15:52.219918 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.210906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:43575 (local address 127.1.253.66:38077)
0924 16:15:52.211597 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:52.211637 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:52.212966 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:52.213487 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:52.213494 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:52.213580 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:52.214289 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.214309 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.215226 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.215230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.216321 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.216331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.217025 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.217028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.217165 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.217840 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:15:52.217844 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:52.217904 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:15:52.218028 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:52.218369 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:52.218380 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:52.218389 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:15:52.218529 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 16:15:52.218853 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":119,"threads_started":1}
I20260924 16:15:52.221148 22302 heartbeater.cc:347] Connected to a master server at 127.1.253.126:43575
I20260924 16:15:52.221386 22302 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:52.221908 22302 heartbeater.cc:511] Master 127.1.253.126:43575 requested a full tablet report, sending...
I20260924 16:15:52.223033 21976 ts_manager.cc:194] Registered new tserver with Master: 5e809f33654647328263c56ce2790bf5 (127.1.253.66:37875)
I20260924 16:15:52.223610 21976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.253.66:38077
W20260924 16:15:52.334414 22305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:52.334707 22305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:52.334787 22305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:52.338879 22305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:52.338972 22305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:52.339062 22305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.253.67
I20260924 16:15:52.344038 22305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.253.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.253.67
--webserver_port=0
--tserver_master_addrs=127.1.253.126:43575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.253.67
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:52.345227 22305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:52.346786 22305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:52.356848 22312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:52.356861 22314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:52.356863 22311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:52.357105 22305 server_base.cc:1034] running on GCE node
I20260924 16:15:52.357636 22305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:52.358423 22305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:52.359596 22305 hybrid_clock.cc:648] HybridClock initialized: now 1790266552359574 us; error 30 us; skew 500 ppm
I20260924 16:15:52.361474 22305 webserver.cc:460] Webserver started at http://127.1.253.67:33983/ using document root <none> and password file <none>
I20260924 16:15:52.362087 22305 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:52.362152 22305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:52.362468 22305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:52.364856 22305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/data/instance:
uuid: "8b14c042b7e646dd8d9da913299af971"
format_stamp: "Formatted at 2026-09-24 16:15:52 on dist-test-slave-6wvm"
server_key: "26a691bc0f634d3044d9150318672dae"
server_key_iv: "2c2482dc339b4eefc968776c9c9fb1f7"
server_key_version: "encryptionkey@0"
I20260924 16:15:52.365515 22305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance:
uuid: "8b14c042b7e646dd8d9da913299af971"
format_stamp: "Formatted at 2026-09-24 16:15:52 on dist-test-slave-6wvm"
server_key: "26a691bc0f634d3044d9150318672dae"
server_key_iv: "2c2482dc339b4eefc968776c9c9fb1f7"
server_key_version: "encryptionkey@0"
I20260924 16:15:52.369468 22305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:15:52.372058 22320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:52.373468 22305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:15:52.373629 22305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/wal
uuid: "8b14c042b7e646dd8d9da913299af971"
format_stamp: "Formatted at 2026-09-24 16:15:52 on dist-test-slave-6wvm"
server_key: "26a691bc0f634d3044d9150318672dae"
server_key_iv: "2c2482dc339b4eefc968776c9c9fb1f7"
server_key_version: "encryptionkey@0"
I20260924 16:15:52.373751 22305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:52.401419 22305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:52.402189 22305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:52.402450 22305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:52.403096 22305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:52.404147 22305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:52.404220 22305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:52.404296 22305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:52.404340 22305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:52.413483 22305 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.67:40295
I20260924 16:15:52.413506 22436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.67:40295 every 8 connection(s)
I20260924 16:15:52.414449 22305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:52.423866 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.416364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.67:46221 (local address 127.1.253.126:43575)
0924 16:15:52.416507 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:52.416511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:52.417594 (+ 1083us) server_negotiation.cc:407] Connection header received
0924 16:15:52.419099 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:52.419103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:52.419167 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:52.419276 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:52.420192 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.420741 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.421898 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.422126 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.422240 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.423360 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:52.423364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:52.423366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:52.423378 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:52.423397 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:52.423404 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:52.423481 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:52.423652 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 16:15:52.423714 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:15:52.424027 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 22305
I20260924 16:15:52.424166 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/wal/instance
I20260924 16:15:52.424496 2037 external_mini_cluster.cc:1428] Setting key 0c8cbb962549671a6ef33f29324d0784
I20260924 16:15:52.424836 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.416778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:43575 (local address 127.1.253.67:46221)
0924 16:15:52.417442 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:52.417483 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:52.418869 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:52.419464 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:52.419471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:52.419558 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:52.420045 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.420057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.420886 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.420889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.421767 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.421776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.422309 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.422314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.422479 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.423083 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:15:52.423085 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:52.423127 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:15:52.423190 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:52.423483 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:52.423496 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:52.423502 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:15:52.423651 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 16:15:52.423925 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260924 16:15:52.425966 22437 heartbeater.cc:347] Connected to a master server at 127.1.253.126:43575
I20260924 16:15:52.426210 22437 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:52.426802 22437 heartbeater.cc:511] Master 127.1.253.126:43575 requested a full tablet report, sending...
I20260924 16:15:52.427834 21976 ts_manager.cc:194] Registered new tserver with Master: 8b14c042b7e646dd8d9da913299af971 (127.1.253.67:40295)
I20260924 16:15:52.428437 21976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.253.67:46221
I20260924 16:15:52.438812 2037 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:15:52.444763 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.440939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.1.253.126:43575)
0924 16:15:52.441066 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:52.441070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:52.441268 (+ 198us) server_negotiation.cc:407] Connection header received
0924 16:15:52.441381 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:52.441383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:52.441424 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:52.441491 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:52.441761 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.442325 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.443163 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.443367 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.443441 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.444317 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:52.444320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:52.444322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:52.444333 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:52.444349 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:52.444356 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:52.444409 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:52.444606 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 16:15:52.444654 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:15:52.447875 21976 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:15:52.451035 21976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:15:52.466080 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.459799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:36707 (local address 127.0.0.1:60616)
0924 16:15:52.460189 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:52.460247 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:52.460383 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:52.461288 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:52.461294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:52.461318 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:52.461450 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.461455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.462913 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.462917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.464492 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.464505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.465493 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.465496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.465610 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.465652 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 16:15:52.465770 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 16:15:52.465914 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":62,"thread_start_us":84,"threads_started":1}
I20260924 16:15:52.466086 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.460117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.1.253.65:36707)
0924 16:15:52.460727 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:52.460740 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:52.460766 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:15:52.460848 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:52.460853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:52.461078 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:52.461246 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:52.461577 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.462793 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.464642 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.465372 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.465491 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.465693 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:52.465800 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 16:15:52.465902 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":157,"threads_started":1}
I20260924 16:15:52.466513 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.459968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.1.253.66:37875)
0924 16:15:52.461275 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:52.461289 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:52.461314 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:15:52.461395 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:52.461400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:52.461604 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:52.461761 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:52.462009 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.463073 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.464701 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.465480 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.465963 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.466151 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:52.466207 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 16:15:52.466334 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":161,"threads_started":1}
I20260924 16:15:52.466113 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.459799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.66:37875 (local address 127.0.0.1:49032)
0924 16:15:52.460704 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:52.460721 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:52.460815 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:52.461755 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:52.461758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:52.461778 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:52.461876 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.461882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.463235 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.463239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.464532 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.464542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.465623 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.465626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.465711 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.465734 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:15:52.465811 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 16:15:52.465957 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":660,"thread_start_us":88,"threads_started":1}
I20260924 16:15:52.466585 22452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.460658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:40295 (local address 127.0.0.1:38958)
0924 16:15:52.461010 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:52.461024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:52.461104 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:52.461936 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:52.461939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:52.461956 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:52.462059 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.462064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.463147 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.463154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.464539 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.464550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.465843 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.465849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.466268 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.466290 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:15:52.466365 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 16:15:52.466462 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":106,"thread_start_us":89,"threads_started":1}
I20260924 16:15:52.467463 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.461093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.1.253.67:40295)
0924 16:15:52.461512 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:52.461522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:52.461542 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:15:52.461612 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:52.461620 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:52.461778 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:52.461911 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:52.462187 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.463021 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.464663 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.465480 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.465597 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.465782 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:52.466400 (+ 618us) server_negotiation.cc:299] Negotiation successful
0924 16:15:52.467313 (+ 913us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":131,"threads_started":1}
I20260924 16:15:52.469456 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 3649a56f03cc4b4eb9f4d74b8d01bdec (DEFAULT_TABLE table=test-table [id=73e6d4258878496caf2fd8df7b28ac23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:52.469465 22234 tablet_service.cc:1467] Processing CreateTablet for tablet 3649a56f03cc4b4eb9f4d74b8d01bdec (DEFAULT_TABLE table=test-table [id=73e6d4258878496caf2fd8df7b28ac23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:52.470475 22234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3649a56f03cc4b4eb9f4d74b8d01bdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:52.470460 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3649a56f03cc4b4eb9f4d74b8d01bdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:52.476855 22454 tablet_bootstrap.cc:492] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4: Bootstrap starting.
I20260924 16:15:52.476552 22369 tablet_service.cc:1467] Processing CreateTablet for tablet 3649a56f03cc4b4eb9f4d74b8d01bdec (DEFAULT_TABLE table=test-table [id=73e6d4258878496caf2fd8df7b28ac23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:52.477707 22369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3649a56f03cc4b4eb9f4d74b8d01bdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:52.478585 22455 tablet_bootstrap.cc:492] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5: Bootstrap starting.
I20260924 16:15:52.479475 22454 tablet_bootstrap.cc:654] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:52.480366 22454 log.cc:824] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:52.480780 22455 tablet_bootstrap.cc:654] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:52.481626 22455 log.cc:824] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:52.482612 22454 tablet_bootstrap.cc:492] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4: No bootstrap required, opened a new log
I20260924 16:15:52.482811 22454 ts_tablet_manager.cc:1397] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:15:52.483811 22455 tablet_bootstrap.cc:492] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5: No bootstrap required, opened a new log
I20260924 16:15:52.483991 22455 ts_tablet_manager.cc:1397] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:15:52.485011 22459 tablet_bootstrap.cc:492] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971: Bootstrap starting.
I20260924 16:15:52.487042 22459 tablet_bootstrap.cc:654] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:52.487282 22454 raft_consensus.cc:348] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e809f33654647328263c56ce2790bf5" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37875 } } peers { permanent_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 36707 } } peers { permanent_uuid: "8b14c042b7e646dd8d9da913299af971" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 40295 } }
I20260924 16:15:52.487564 22454 raft_consensus.cc:374] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:52.487651 22454 raft_consensus.cc:729] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1fdc1ca55d49b6a55a98d5bf2d1fb4, State: Initialized, Role: FOLLOWER
I20260924 16:15:52.488155 22454 consensus_queue.cc:260] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e809f33654647328263c56ce2790bf5" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37875 } } peers { permanent_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 36707 } } peers { permanent_uuid: "8b14c042b7e646dd8d9da913299af971" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 40295 } }
I20260924 16:15:52.488291 22459 log.cc:824] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:52.488494 22455 raft_consensus.cc:348] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e809f33654647328263c56ce2790bf5" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37875 } } peers { permanent_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 36707 } } peers { permanent_uuid: "8b14c042b7e646dd8d9da913299af971" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 40295 } }
I20260924 16:15:52.488735 22455 raft_consensus.cc:374] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:52.488797 22455 raft_consensus.cc:729] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e809f33654647328263c56ce2790bf5, State: Initialized, Role: FOLLOWER
I20260924 16:15:52.489238 22454 ts_tablet_manager.cc:1428] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 16:15:52.489305 22168 heartbeater.cc:503] Master 127.1.253.126:43575 was elected leader, sending a full tablet report...
I20260924 16:15:52.489296 22455 consensus_queue.cc:260] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e809f33654647328263c56ce2790bf5" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37875 } } peers { permanent_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 36707 } } peers { permanent_uuid: "8b14c042b7e646dd8d9da913299af971" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 40295 } }
I20260924 16:15:52.490026 22302 heartbeater.cc:503] Master 127.1.253.126:43575 was elected leader, sending a full tablet report...
I20260924 16:15:52.490525 22459 tablet_bootstrap.cc:492] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971: No bootstrap required, opened a new log
I20260924 16:15:52.490738 22459 ts_tablet_manager.cc:1397] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 16:15:52.491047 22455 ts_tablet_manager.cc:1428] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 16:15:52.493430 22462 raft_consensus.cc:482] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:52.493636 22462 raft_consensus.cc:504] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e809f33654647328263c56ce2790bf5" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37875 } } peers { permanent_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 36707 } } peers { permanent_uuid: "8b14c042b7e646dd8d9da913299af971" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 40295 } }
I20260924 16:15:52.495016 22462 leader_election.cc:288] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1fdc1ca55d49b6a55a98d5bf2d1fb4 (127.1.253.65:36707), 8b14c042b7e646dd8d9da913299af971 (127.1.253.67:40295)
I20260924 16:15:52.495635 22459 raft_consensus.cc:348] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e809f33654647328263c56ce2790bf5" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37875 } } peers { permanent_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 36707 } } peers { permanent_uuid: "8b14c042b7e646dd8d9da913299af971" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 40295 } }
I20260924 16:15:52.495920 22459 raft_consensus.cc:374] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:52.495990 22459 raft_consensus.cc:729] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b14c042b7e646dd8d9da913299af971, State: Initialized, Role: FOLLOWER
I20260924 16:15:52.496462 22459 consensus_queue.cc:260] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e809f33654647328263c56ce2790bf5" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37875 } } peers { permanent_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 36707 } } peers { permanent_uuid: "8b14c042b7e646dd8d9da913299af971" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 40295 } }
I20260924 16:15:52.497154 22437 heartbeater.cc:503] Master 127.1.253.126:43575 was elected leader, sending a full tablet report...
I20260924 16:15:52.497568 22459 ts_tablet_manager.cc:1428] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 16:15:52.499936 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.495185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:36707 (local address 127.1.253.66:46619)
0924 16:15:52.495299 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:52.495317 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:52.495405 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:52.496122 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:52.496125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:52.496151 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:52.496327 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.496335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.497330 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.497333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.498532 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.498543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.499508 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.499510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.499588 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.499641 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 16:15:52.499713 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 16:15:52.499781 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":35}
I20260924 16:15:52.500289 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.495244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:46619 (local address 127.1.253.65:36707)
0924 16:15:52.495370 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:52.495374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:52.495391 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:15:52.495541 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:52.495544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:52.495596 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:52.495673 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:52.496452 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.497195 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.498678 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.499291 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.499399 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.499455 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:52.500088 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 16:15:52.500176 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:15:52.500705 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.495587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.253.66:48079 (local address 127.1.253.67:40295)
0924 16:15:52.495819 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:52.495822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:52.495832 (+ 10us) server_negotiation.cc:407] Connection header received
0924 16:15:52.495872 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:52.495874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:52.495906 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:52.496001 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:52.497329 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.497969 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.499333 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.499922 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.500067 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.500110 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:52.500476 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 16:15:52.500553 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260924 16:15:52.500921 22121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3649a56f03cc4b4eb9f4d74b8d01bdec" candidate_uuid: "5e809f33654647328263c56ce2790bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4" is_pre_election: true
I20260924 16:15:52.501118 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.495217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.67:40295 (local address 127.1.253.66:48079)
0924 16:15:52.495527 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:52.495540 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:52.495615 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:52.496036 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:52.496040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:52.496064 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:52.497188 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.497200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.498088 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.498090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.499183 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:52.499194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.500229 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.500232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:52.500311 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.500328 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 16:15:52.500873 (+ 545us) client_negotiation.cc:240] Negotiation successful
0924 16:15:52.500963 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":23,"thread_start_us":108,"threads_started":1}
I20260924 16:15:52.501220 22121 raft_consensus.cc:2454] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e809f33654647328263c56ce2790bf5 in term 0.
I20260924 16:15:52.501703 22389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3649a56f03cc4b4eb9f4d74b8d01bdec" candidate_uuid: "5e809f33654647328263c56ce2790bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b14c042b7e646dd8d9da913299af971" is_pre_election: true
I20260924 16:15:52.501713 22189 leader_election.cc:302] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1fdc1ca55d49b6a55a98d5bf2d1fb4, 5e809f33654647328263c56ce2790bf5; no voters:
I20260924 16:15:52.501945 22389 raft_consensus.cc:2454] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e809f33654647328263c56ce2790bf5 in term 0.
I20260924 16:15:52.502044 22462 raft_consensus.cc:2781] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:15:52.502149 22462 raft_consensus.cc:482] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:15:52.502224 22462 raft_consensus.cc:3037] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:52.503294 22462 raft_consensus.cc:504] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e809f33654647328263c56ce2790bf5" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37875 } } peers { permanent_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 36707 } } peers { permanent_uuid: "8b14c042b7e646dd8d9da913299af971" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 40295 } }
I20260924 16:15:52.503674 22462 leader_election.cc:288] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1fdc1ca55d49b6a55a98d5bf2d1fb4 (127.1.253.65:36707), 8b14c042b7e646dd8d9da913299af971 (127.1.253.67:40295)
I20260924 16:15:52.504000 22389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3649a56f03cc4b4eb9f4d74b8d01bdec" candidate_uuid: "5e809f33654647328263c56ce2790bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b14c042b7e646dd8d9da913299af971"
I20260924 16:15:52.504051 22121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3649a56f03cc4b4eb9f4d74b8d01bdec" candidate_uuid: "5e809f33654647328263c56ce2790bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4"
I20260924 16:15:52.504104 22389 raft_consensus.cc:3037] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:52.504181 22121 raft_consensus.cc:3037] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:52.505112 22389 raft_consensus.cc:2454] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e809f33654647328263c56ce2790bf5 in term 1.
I20260924 16:15:52.505191 22121 raft_consensus.cc:2454] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e809f33654647328263c56ce2790bf5 in term 1.
I20260924 16:15:52.505457 22188 leader_election.cc:302] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e809f33654647328263c56ce2790bf5, 8b14c042b7e646dd8d9da913299af971; no voters:
I20260924 16:15:52.505650 22462 raft_consensus.cc:2781] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:52.505903 22462 raft_consensus.cc:686] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 1 LEADER]: Becoming Leader. State: Replica: 5e809f33654647328263c56ce2790bf5, State: Running, Role: LEADER
I20260924 16:15:52.506217 22462 consensus_queue.cc:237] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e809f33654647328263c56ce2790bf5" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37875 } } peers { permanent_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 36707 } } peers { permanent_uuid: "8b14c042b7e646dd8d9da913299af971" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 40295 } }
I20260924 16:15:52.509449 21976 catalog_manager.cc:5495] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e809f33654647328263c56ce2790bf5 (127.1.253.66). New cstate: current_term: 1 leader_uuid: "5e809f33654647328263c56ce2790bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e809f33654647328263c56ce2790bf5" member_type: VOTER last_known_addr { host: "127.1.253.66" port: 37875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 36707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b14c042b7e646dd8d9da913299af971" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 40295 } health_report { overall_health: UNKNOWN } } }
W20260924 16:15:52.509893 22169 tablet.cc:2369] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:15:52.528791 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.525158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.1.253.66:37875)
0924 16:15:52.525298 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:52.525302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:52.525318 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:15:52.525434 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:52.525436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:52.525499 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:52.525603 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:15:52.525837 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.526445 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.527291 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:52.527531 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:52.527627 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:52.527946 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:15:52.528393 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:15:52.528497 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:52.528548 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 16:15:52.528601 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:15:52.535130 22389 raft_consensus.cc:1261] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971 [term 1 FOLLOWER]: Refusing update from remote peer 5e809f33654647328263c56ce2790bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:52.535180 22121 raft_consensus.cc:1261] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 3d1fdc1ca55d49b6a55a98d5bf2d1fb4 [term 1 FOLLOWER]: Refusing update from remote peer 5e809f33654647328263c56ce2790bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:52.535884 22462 consensus_queue.cc:1035] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b14c042b7e646dd8d9da913299af971" member_type: VOTER last_known_addr { host: "127.1.253.67" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:52.536135 22465 consensus_queue.cc:1035] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1fdc1ca55d49b6a55a98d5bf2d1fb4" member_type: VOTER last_known_addr { host: "127.1.253.65" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:52.543490 22468 mvcc.cc:205] Tried to move back new op lower bound from 7332931799175462912 to 7332931799068536832. Current Snapshot: MvccSnapshot[applied={T|T < 7332931799175462912}]
I20260924 16:15:52.546061 22471 mvcc.cc:205] Tried to move back new op lower bound from 7332931799175462912 to 7332931799068536832. Current Snapshot: MvccSnapshot[applied={T|T < 7332931799175462912}]
I20260924 16:15:52.553684 21975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:52.553875 21975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:52.556545 21975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 460a5de8d34148a8994ea678f16701cf: Sending DeleteTablet for 3 replicas of tablet 3649a56f03cc4b4eb9f4d74b8d01bdec
I20260924 16:15:52.557441 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 3649a56f03cc4b4eb9f4d74b8d01bdec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:52 UTC) from {username='slave'} at 127.0.0.1:60616
I20260924 16:15:52.557590 22234 tablet_service.cc:1514] Processing DeleteTablet for tablet 3649a56f03cc4b4eb9f4d74b8d01bdec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:52 UTC) from {username='slave'} at 127.0.0.1:49032
I20260924 16:15:52.557746 22369 tablet_service.cc:1514] Processing DeleteTablet for tablet 3649a56f03cc4b4eb9f4d74b8d01bdec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:52 UTC) from {username='slave'} at 127.0.0.1:38958
I20260924 16:15:52.558142 22478 tablet_replica.cc:330] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5: stopping tablet replica
I20260924 16:15:52.558183 22479 tablet_replica.cc:330] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971: stopping tablet replica
I20260924 16:15:52.558559 22479 raft_consensus.cc:2229] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:52.558602 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 22037
I20260924 16:15:52.558568 22478 raft_consensus.cc:2229] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:15:52.560695 22478 raft_consensus.cc:2258] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:52.560858 22479 raft_consensus.cc:2258] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:52.562628 22478 ts_tablet_manager.cc:1905] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:52.562629 22479 ts_tablet_manager.cc:1905] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:52.565615 22478 ts_tablet_manager.cc:1918] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:52.566020 22478 log.cc:1196] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-1/wal/wals/3649a56f03cc4b4eb9f4d74b8d01bdec
I20260924 16:15:52.566303 22479 ts_tablet_manager.cc:1918] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:52.566412 22479 log.cc:1196] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971: Deleting WAL directory at /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266297063660-2037-0/minicluster-data/ts-2/wal/wals/3649a56f03cc4b4eb9f4d74b8d01bdec
I20260924 16:15:52.566582 22478 ts_tablet_manager.cc:1939] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 5e809f33654647328263c56ce2790bf5: Deleting consensus metadata
I20260924 16:15:52.566748 22479 ts_tablet_manager.cc:1939] T 3649a56f03cc4b4eb9f4d74b8d01bdec P 8b14c042b7e646dd8d9da913299af971: Deleting consensus metadata
I20260924 16:15:52.568013 21960 catalog_manager.cc:4841] TS 5e809f33654647328263c56ce2790bf5 (127.1.253.66:37875): tablet 3649a56f03cc4b4eb9f4d74b8d01bdec (table test-table [id=73e6d4258878496caf2fd8df7b28ac23]) successfully deleted
I20260924 16:15:52.568253 21960 catalog_manager.cc:4841] TS 8b14c042b7e646dd8d9da913299af971 (127.1.253.67:40295): tablet 3649a56f03cc4b4eb9f4d74b8d01bdec (table test-table [id=73e6d4258878496caf2fd8df7b28ac23]) successfully deleted
W20260924 16:15:52.570086 21961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.253.65:36707 (error 108)
I20260924 16:15:52.570925 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 22171
I20260924 16:15:52.571265 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.570941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.65:36707 (local address 127.0.0.1:60626)
0924 16:15:52.571088 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:52.571166 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.65:36707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:15:52.573374 21961 catalog_manager.cc:4584] TS 3d1fdc1ca55d49b6a55a98d5bf2d1fb4 (127.1.253.65:36707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3649a56f03cc4b4eb9f4d74b8d01bdec: Network error: Client connection negotiation failed: client connection to 127.1.253.65:36707: connect: Connection refused (error 111)
I20260924 16:15:52.578426 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 22305
I20260924 16:15:52.584643 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 21944
2026-09-24T16:15:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1070 ms)
[----------] 6 tests from EncryptionPolicyTest (10646 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:52 dist-test-slave-6wvm krb5kdc[22483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:52 dist-test-slave-6wvm krb5kdc[22483](info): set up 2 sockets
Sep 24 16:15:52 dist-test-slave-6wvm krb5kdc[22483](info): commencing operation
krb5kdc: starting...
W20260924 16:15:53.629947 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:15:53 dist-test-slave-6wvm krb5kdc[22483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:15:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:15:53.761363 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:53.881502 22496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:53.881774 22496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:53.881816 22496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:53.885883 22496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:53.885967 22496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:53.885985 22496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:53.886003 22496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:53.886016 22496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:53.891638 22496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:53.892876 22496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:53.894019 22496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:53.902787 22501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:53.902779 22502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:53.902787 22504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:53.903055 22496 server_base.cc:1034] running on GCE node
I20260924 16:15:53.903689 22496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:53.904843 22496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:53.906010 22496 hybrid_clock.cc:648] HybridClock initialized: now 1790266553905990 us; error 32 us; skew 500 ppm
Sep 24 16:15:53 dist-test-slave-6wvm krb5kdc[22483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:53.917529 22496 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:15:53.918609 22496 webserver.cc:460] Webserver started at http://127.0.0.1:45447/ using document root <none> and password file <none>
I20260924 16:15:53.919198 22496 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:53.919292 22496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:53.919543 22496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:53.921684 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "3c2402af932e457588e6e50743f48dbe"
format_stamp: "Formatted at 2026-09-24 16:15:53 on dist-test-slave-6wvm"
server_key: "1da63f7b7365117448702dc6d0fbc184"
server_key_iv: "709cbbeabe76c92f0e8efba38e87a7a4"
server_key_version: "encryptionkey@0"
I20260924 16:15:53.922322 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "3c2402af932e457588e6e50743f48dbe"
format_stamp: "Formatted at 2026-09-24 16:15:53 on dist-test-slave-6wvm"
server_key: "1da63f7b7365117448702dc6d0fbc184"
server_key_iv: "709cbbeabe76c92f0e8efba38e87a7a4"
server_key_version: "encryptionkey@0"
I20260924 16:15:53.925978 22496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:53.928763 22511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:53.930402 22496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:15:53.930550 22496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "3c2402af932e457588e6e50743f48dbe"
format_stamp: "Formatted at 2026-09-24 16:15:53 on dist-test-slave-6wvm"
server_key: "1da63f7b7365117448702dc6d0fbc184"
server_key_iv: "709cbbeabe76c92f0e8efba38e87a7a4"
server_key_version: "encryptionkey@0"
I20260924 16:15:53.930667 22496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:53.953510 22496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:53.957739 22496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:53.958010 22496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:53.966611 22496 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33549
I20260924 16:15:53.966643 22564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33549 every 8 connection(s)
I20260924 16:15:53.967790 22496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:53.968925 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 22496
I20260924 16:15:53.969100 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:15:53.969451 2037 external_mini_cluster.cc:1428] Setting key 378c1551594f3b5e625a07ecfad1ebae
I20260924 16:15:53.971318 22565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:53.978637 22565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe: Bootstrap starting.
Sep 24 16:15:53 dist-test-slave-6wvm krb5kdc[22483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:15:53.981459 22565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:53.982475 22565 log.cc:824] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:53.985095 22565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe: No bootstrap required, opened a new log
I20260924 16:15:53.986459 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:53.971094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35085 (local address 127.0.0.1:33549)
0924 16:15:53.971564 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:53.971584 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:53.971833 (+ 249us) server_negotiation.cc:407] Connection header received
0924 16:15:53.972792 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:53.972815 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:53.973276 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:53.973659 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:53.974599 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:53.976120 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:53.977022 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:53.977735 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:53.977897 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:53.980754 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:53.980760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:53.980778 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:53.980811 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:53.982848 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:53.983600 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:53.983606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:53.983612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:53.983745 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:53.984120 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:53.984124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:53.984125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:53.984631 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:53.984838 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:53.985201 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 16:15:53.985447 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":136,"threads_started":1}
I20260924 16:15:53.989189 22565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2402af932e457588e6e50743f48dbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33549 } }
I20260924 16:15:53.989425 22565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:53.989506 22565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2402af932e457588e6e50743f48dbe, State: Initialized, Role: FOLLOWER
I20260924 16:15:53.989972 22565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2402af932e457588e6e50743f48dbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33549 } }
I20260924 16:15:53.990129 22565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:53.990185 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:15:53.990221 22565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:53.990352 22565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:53.991326 22565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2402af932e457588e6e50743f48dbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33549 } }
I20260924 16:15:53.991704 22565 leader_election.cc:302] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c2402af932e457588e6e50743f48dbe; no voters:
I20260924 16:15:53.992090 22565 leader_election.cc:288] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:53.992168 22575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:53.992383 22575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [term 1 LEADER]: Becoming Leader. State: Replica: 3c2402af932e457588e6e50743f48dbe, State: Running, Role: LEADER
I20260924 16:15:53.992712 22575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2402af932e457588e6e50743f48dbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33549 } }
I20260924 16:15:53.993615 22565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:53.994639 22577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c2402af932e457588e6e50743f48dbe. Latest consensus state: current_term: 1 leader_uuid: "3c2402af932e457588e6e50743f48dbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2402af932e457588e6e50743f48dbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33549 } } }
I20260924 16:15:53.994771 22577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:53.995038 22576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c2402af932e457588e6e50743f48dbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2402af932e457588e6e50743f48dbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33549 } } }
I20260924 16:15:53.995147 22576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe [sys.catalog]: This master's current role is: LEADER
W20260924 16:15:53.998704 22592 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:15:53.998814 22592 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:15:53.998881 22585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:53.999853 22585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:54.003162 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:53.992406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45173 (local address 127.0.0.1:33549)
0924 16:15:53.992520 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:53.992525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:53.993306 (+ 781us) server_negotiation.cc:407] Connection header received
0924 16:15:53.993424 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:53.993427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:53.993477 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:53.993563 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:53.994546 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:53.995383 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:53.996522 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:53.996975 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:53.997080 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:53.998868 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:53.998872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:53.998877 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:53.998904 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:54.000691 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:54.001299 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:54.001307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:54.001309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:54.001399 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:54.001817 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:54.001821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:54.001822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:54.002170 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:54.002436 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:54.002785 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 16:15:54.002949 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:15:54.006412 22585 catalog_manager.cc:1347] Generated new cluster ID: 3b55d161b9b44a9e86cbab93fcb7d210
I20260924 16:15:54.006488 22585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:54.027343 22585 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:54.028687 22585 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:54.038043 22585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c2402af932e457588e6e50743f48dbe: Generated new TSK 0
I20260924 16:15:54.038918 22585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:15:54.050655 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 22496
W20260924 16:15:54.057160 22481 connection.cc:535] client connection to 127.0.0.1:33549 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33549): Connection reset by peer (error 104)
2026-09-24T16:15:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1464 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:54 dist-test-slave-6wvm krb5kdc[22598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:54 dist-test-slave-6wvm krb5kdc[22598](info): set up 2 sockets
Sep 24 16:15:54 dist-test-slave-6wvm krb5kdc[22598](info): commencing operation
krb5kdc: starting...
W20260924 16:15:55.125890 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:15:55 dist-test-slave-6wvm krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:15:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:15:55.253845 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:55.365653 22612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:55.365911 22612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:55.365957 22612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:55.369800 22612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:55.369849 22612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:55.369867 22612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:55.369884 22612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:55.369899 22612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:55.375100 22612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:55.376075 22612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:55.377046 22612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:55.385422 22621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:55.385416 22617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:55.385416 22619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:55.385651 22612 server_base.cc:1034] running on GCE node
I20260924 16:15:55.386328 22612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:55.387439 22612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:55.388644 22612 hybrid_clock.cc:648] HybridClock initialized: now 1790266555388634 us; error 28 us; skew 500 ppm
Sep 24 16:15:55 dist-test-slave-6wvm krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:55.391489 22612 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:15:55.392318 22612 webserver.cc:460] Webserver started at http://127.0.0.1:39205/ using document root <none> and password file <none>
I20260924 16:15:55.392839 22612 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:55.392896 22612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:55.393064 22612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:55.395136 22612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "7c20e2c5bf694078a076dbdfd54b828e"
format_stamp: "Formatted at 2026-09-24 16:15:55 on dist-test-slave-6wvm"
server_key: "a071223f572c5128292fc32e7efd5e6f"
server_key_iv: "ad6a3afeecc084370f5411236d9e016a"
server_key_version: "encryptionkey@0"
I20260924 16:15:55.395661 22612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "7c20e2c5bf694078a076dbdfd54b828e"
format_stamp: "Formatted at 2026-09-24 16:15:55 on dist-test-slave-6wvm"
server_key: "a071223f572c5128292fc32e7efd5e6f"
server_key_iv: "ad6a3afeecc084370f5411236d9e016a"
server_key_version: "encryptionkey@0"
I20260924 16:15:55.399319 22612 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:15:55.401803 22628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:55.403103 22612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:15:55.403206 22612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "7c20e2c5bf694078a076dbdfd54b828e"
format_stamp: "Formatted at 2026-09-24 16:15:55 on dist-test-slave-6wvm"
server_key: "a071223f572c5128292fc32e7efd5e6f"
server_key_iv: "ad6a3afeecc084370f5411236d9e016a"
server_key_version: "encryptionkey@0"
I20260924 16:15:55.403294 22612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:55.418556 22612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:55.422034 22612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:55.422196 22612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:55.429522 22612 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44803
I20260924 16:15:55.429550 22680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44803 every 8 connection(s)
I20260924 16:15:55.430609 22612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:55.433434 22681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:55.439253 22681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e: Bootstrap starting.
I20260924 16:15:55.440398 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 22612
I20260924 16:15:55.440537 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:15:55.440847 2037 external_mini_cluster.cc:1428] Setting key 8a5b08157d067b020305e90454d77445
I20260924 16:15:55.442181 22681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:55.443135 22681 log.cc:824] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:55.445333 22681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e: No bootstrap required, opened a new log
I20260924 16:15:55.449481 22681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c20e2c5bf694078a076dbdfd54b828e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44803 } }
I20260924 16:15:55.449756 22681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:55.449854 22681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c20e2c5bf694078a076dbdfd54b828e, State: Initialized, Role: FOLLOWER
I20260924 16:15:55.450423 22681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c20e2c5bf694078a076dbdfd54b828e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44803 } }
I20260924 16:15:55.450600 22681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:55.450697 22681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:55.450798 22681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:55.451126 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:55.442128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44291 (local address 127.0.0.1:44803)
0924 16:15:55.442650 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:55.442673 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:55.442715 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:15:55.443616 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:55.443638 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:55.444040 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:55.444400 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:55.444824 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:55.446312 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:55.447198 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:55.447935 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:55.448128 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:55.448980 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:55.448987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:55.449004 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:55.449515 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:55.449585 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:55.449617 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:55.449755 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:55.449990 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 16:15:55.450155 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":152,"threads_started":1}
I20260924 16:15:55.451898 22681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c20e2c5bf694078a076dbdfd54b828e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44803 } }
I20260924 16:15:55.452270 22681 leader_election.cc:302] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c20e2c5bf694078a076dbdfd54b828e; no voters:
I20260924 16:15:55.452606 22681 leader_election.cc:288] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:55.452684 22686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:55.452865 22686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [term 1 LEADER]: Becoming Leader. State: Replica: 7c20e2c5bf694078a076dbdfd54b828e, State: Running, Role: LEADER
I20260924 16:15:55.453171 22686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c20e2c5bf694078a076dbdfd54b828e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44803 } }
I20260924 16:15:55.453886 22681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:55.454589 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:15:55.454871 22687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c20e2c5bf694078a076dbdfd54b828e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c20e2c5bf694078a076dbdfd54b828e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44803 } } }
I20260924 16:15:55.455015 22687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:55.455331 22688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c20e2c5bf694078a076dbdfd54b828e. Latest consensus state: current_term: 1 leader_uuid: "7c20e2c5bf694078a076dbdfd54b828e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c20e2c5bf694078a076dbdfd54b828e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44803 } } }
I20260924 16:15:55.455431 22688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:55.455768 22695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:55.459064 22695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:55.461925 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:55.457062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47651 (local address 127.0.0.1:44803)
0924 16:15:55.457180 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:55.457186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:55.457429 (+ 243us) server_negotiation.cc:407] Connection header received
0924 16:15:55.457553 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:55.457555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:55.457633 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:55.457714 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:55.458055 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:55.458851 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:55.459964 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:55.460304 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:55.460418 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:55.461407 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:55.461411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:55.461416 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:55.461433 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:55.461461 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:55.461475 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:55.461570 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:55.461748 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 16:15:55.461784 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 16:15:55.464857 22695 catalog_manager.cc:1347] Generated new cluster ID: 21f5b967723d4e0b9ac109fa562f9752
I20260924 16:15:55.464920 22695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:55.484751 22695 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:55.486440 22695 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:55.494498 22695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c20e2c5bf694078a076dbdfd54b828e: Generated new TSK 0
I20260924 16:15:55.495237 22695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:15:55.499282 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 22612
W20260924 16:15:55.505992 22596 connection.cc:535] client connection to 127.0.0.1:44803 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44803): Connection reset by peer (error 104)
2026-09-24T16:15:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:55 dist-test-slave-6wvm krb5kdc[22713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:55 dist-test-slave-6wvm krb5kdc[22713](info): set up 2 sockets
Sep 24 16:15:55 dist-test-slave-6wvm krb5kdc[22713](info): commencing operation
krb5kdc: starting...
W20260924 16:15:56.533952 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:15:56 dist-test-slave-6wvm krb5kdc[22713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:15:56.661417 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:56.777559 22726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:56.777807 22726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:56.777846 22726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:56.781801 22726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:56.781857 22726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:56.781889 22726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:56.781904 22726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:56.781919 22726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:56.787086 22726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:56.788136 22726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:56.789155 22726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:56.797997 22732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:56.798050 22731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:56.798007 22734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:56.798322 22726 server_base.cc:1034] running on GCE node
I20260924 16:15:56.798931 22726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:56.800081 22726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:56.801275 22726 hybrid_clock.cc:648] HybridClock initialized: now 1790266556801258 us; error 29 us; skew 500 ppm
Sep 24 16:15:56 dist-test-slave-6wvm krb5kdc[22713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:56.804090 22726 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:15:56.805049 22726 webserver.cc:460] Webserver started at http://127.0.0.1:46213/ using document root <none> and password file <none>
I20260924 16:15:56.805635 22726 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:56.805727 22726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:56.805967 22726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:56.808166 22726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "5d8292c6148b4dc0ab179ea4c7f2a829"
format_stamp: "Formatted at 2026-09-24 16:15:56 on dist-test-slave-6wvm"
server_key: "782f17c944c95a3df17b405fe367bda5"
server_key_iv: "c19fe228bc40621697b0e211e2e1e822"
server_key_version: "encryptionkey@0"
I20260924 16:15:56.808763 22726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "5d8292c6148b4dc0ab179ea4c7f2a829"
format_stamp: "Formatted at 2026-09-24 16:15:56 on dist-test-slave-6wvm"
server_key: "782f17c944c95a3df17b405fe367bda5"
server_key_iv: "c19fe228bc40621697b0e211e2e1e822"
server_key_version: "encryptionkey@0"
I20260924 16:15:56.812697 22726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:15:56.815335 22741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:56.816579 22726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:15:56.816731 22726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "5d8292c6148b4dc0ab179ea4c7f2a829"
format_stamp: "Formatted at 2026-09-24 16:15:56 on dist-test-slave-6wvm"
server_key: "782f17c944c95a3df17b405fe367bda5"
server_key_iv: "c19fe228bc40621697b0e211e2e1e822"
server_key_version: "encryptionkey@0"
I20260924 16:15:56.816855 22726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:56.821703 22726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:56.824623 22726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:56.824817 22726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:56.831620 22726 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39457
I20260924 16:15:56.831698 22794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39457 every 8 connection(s)
I20260924 16:15:56.832574 22726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:56.835448 22795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:56.838320 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 22726
I20260924 16:15:56.838492 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:15:56.838796 2037 external_mini_cluster.cc:1428] Setting key 52053de36ee37017db516a75c94d978f
I20260924 16:15:56.841769 22795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829: Bootstrap starting.
I20260924 16:15:56.844208 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:56.840277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59895 (local address 127.0.0.1:39457)
0924 16:15:56.840766 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:56.840781 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:56.840817 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:15:56.841465 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:56.841484 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:56.841789 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:56.842178 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:56.843005 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:56.843009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:56.843021 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:56.843034 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:56.843091 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:56.843111 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:56.843265 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:56.843338 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 16:15:56.843504 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":159,"threads_started":1}
I20260924 16:15:56.844396 22795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:56.845163 22795 log.cc:824] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:56.847203 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:15:56.847434 22795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829: No bootstrap required, opened a new log
I20260924 16:15:56.851140 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:56.849348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42439 (local address 127.0.0.1:39457)
0924 16:15:56.849470 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:56.849475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:56.849620 (+ 145us) server_negotiation.cc:407] Connection header received
0924 16:15:56.849755 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:56.849757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:56.849789 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:56.849845 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:56.850704 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:56.850707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:56.850709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:56.850719 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:56.850746 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:56.850756 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:56.850819 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:56.850938 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 16:15:56.850973 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:15:56.851897 22795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8292c6148b4dc0ab179ea4c7f2a829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39457 } }
I20260924 16:15:56.852200 22795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:56.852265 22795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d8292c6148b4dc0ab179ea4c7f2a829, State: Initialized, Role: FOLLOWER
I20260924 16:15:56.852675 22795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8292c6148b4dc0ab179ea4c7f2a829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39457 } }
I20260924 16:15:56.852792 22795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:56.852833 22795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:56.852906 22795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:56.853807 22795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8292c6148b4dc0ab179ea4c7f2a829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39457 } }
I20260924 16:15:56.854121 22795 leader_election.cc:302] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d8292c6148b4dc0ab179ea4c7f2a829; no voters:
I20260924 16:15:56.854445 22795 leader_election.cc:288] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:56.854578 22805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:56.854806 22805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [term 1 LEADER]: Becoming Leader. State: Replica: 5d8292c6148b4dc0ab179ea4c7f2a829, State: Running, Role: LEADER
I20260924 16:15:56.855156 22805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8292c6148b4dc0ab179ea4c7f2a829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39457 } }
I20260924 16:15:56.855803 22795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:56.856946 22807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d8292c6148b4dc0ab179ea4c7f2a829. Latest consensus state: current_term: 1 leader_uuid: "5d8292c6148b4dc0ab179ea4c7f2a829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8292c6148b4dc0ab179ea4c7f2a829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39457 } } }
I20260924 16:15:56.856986 22806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d8292c6148b4dc0ab179ea4c7f2a829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8292c6148b4dc0ab179ea4c7f2a829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39457 } } }
I20260924 16:15:56.857092 22807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:56.857092 22806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:56.858335 22814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:56.861720 22814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:56.867738 22814 catalog_manager.cc:1347] Generated new cluster ID: f603a4fd90e7432ca9d5e3a55ba0c281
I20260924 16:15:56.867830 22814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:56.879309 22814 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:56.879704 22814 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:56.885087 22814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d8292c6148b4dc0ab179ea4c7f2a829: Generated new TSK 0
I20260924 16:15:56.886014 22814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:15:56.888978 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 22726
2026-09-24T16:15:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:56 dist-test-slave-6wvm krb5kdc[22826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:56 dist-test-slave-6wvm krb5kdc[22826](info): set up 2 sockets
Sep 24 16:15:56 dist-test-slave-6wvm krb5kdc[22826](info): commencing operation
krb5kdc: starting...
W20260924 16:15:57.950312 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:15:58 dist-test-slave-6wvm krb5kdc[22826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:15:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:15:58.075691 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:58.192734 22839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:58.192997 22839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:58.193039 22839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:58.196951 22839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:58.197007 22839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:58.197026 22839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:58.197042 22839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:58.197057 22839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:58.202211 22839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:58.203264 22839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:58.204274 22839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:58.213544 22844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:58.213558 22847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:58.213631 22839 server_base.cc:1034] running on GCE node
W20260924 16:15:58.213560 22845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:58.214551 22839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:58.215797 22839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:58.217007 22839 hybrid_clock.cc:648] HybridClock initialized: now 1790266558216976 us; error 44 us; skew 500 ppm
Sep 24 16:15:58 dist-test-slave-6wvm krb5kdc[22826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:58.219944 22839 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:15:58.220924 22839 webserver.cc:460] Webserver started at http://127.0.0.1:42911/ using document root <none> and password file <none>
I20260924 16:15:58.221549 22839 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:58.221647 22839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:58.221944 22839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:58.224159 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "b846637c584c420ab1325d5fd0bed9ae"
format_stamp: "Formatted at 2026-09-24 16:15:58 on dist-test-slave-6wvm"
server_key: "b7e561be29c5431b3d63d3047a87c72c"
server_key_iv: "783f96465cb2b9bbbf36fa1776c892e2"
server_key_version: "encryptionkey@0"
I20260924 16:15:58.224745 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "b846637c584c420ab1325d5fd0bed9ae"
format_stamp: "Formatted at 2026-09-24 16:15:58 on dist-test-slave-6wvm"
server_key: "b7e561be29c5431b3d63d3047a87c72c"
server_key_iv: "783f96465cb2b9bbbf36fa1776c892e2"
server_key_version: "encryptionkey@0"
I20260924 16:15:58.228615 22839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:15:58.231208 22854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:58.232571 22839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 16:15:58.232743 22839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "b846637c584c420ab1325d5fd0bed9ae"
format_stamp: "Formatted at 2026-09-24 16:15:58 on dist-test-slave-6wvm"
server_key: "b7e561be29c5431b3d63d3047a87c72c"
server_key_iv: "783f96465cb2b9bbbf36fa1776c892e2"
server_key_version: "encryptionkey@0"
I20260924 16:15:58.232864 22839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:58.254444 22839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:58.257606 22839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:58.257828 22839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:58.266047 22839 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41487
I20260924 16:15:58.266074 22910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41487 every 8 connection(s)
I20260924 16:15:58.267169 22839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:58.270138 22911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:58.272888 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 22839
I20260924 16:15:58.273064 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:15:58.273344 2037 external_mini_cluster.cc:1428] Setting key 9dcf4b9403ef69311749f92e50aded06
I20260924 16:15:58.276647 22911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae: Bootstrap starting.
I20260924 16:15:58.279364 22911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:58.280220 22911 log.cc:824] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:58.282951 22911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae: No bootstrap required, opened a new log
Sep 24 16:15:58 dist-test-slave-6wvm krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:15:58.286226 22911 raft_consensus.cc:348] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b846637c584c420ab1325d5fd0bed9ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41487 } }
I20260924 16:15:58.286580 22911 raft_consensus.cc:374] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:58.286672 22911 raft_consensus.cc:729] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b846637c584c420ab1325d5fd0bed9ae, State: Initialized, Role: FOLLOWER
I20260924 16:15:58.287216 22911 consensus_queue.cc:260] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b846637c584c420ab1325d5fd0bed9ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41487 } }
I20260924 16:15:58.287398 22911 raft_consensus.cc:388] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:58.287479 22911 raft_consensus.cc:482] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:58.287626 22911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:58.288754 22911 raft_consensus.cc:504] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b846637c584c420ab1325d5fd0bed9ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41487 } }
I20260924 16:15:58.288908 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:58.274927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33157 (local address 127.0.0.1:41487)
0924 16:15:58.275423 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:58.275439 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:58.275477 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:15:58.276115 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:58.276133 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:58.276492 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:58.276859 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:58.277750 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:58.278894 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:58.279835 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:58.280317 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:58.280490 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:58.283595 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:58.283601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:58.283615 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:58.283644 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:58.285914 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:58.286459 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:58.286464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:58.286469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:58.286610 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:58.286882 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:58.286885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:58.286887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:58.287303 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:58.287537 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:58.287797 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 16:15:58.288104 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":155,"threads_started":1}
I20260924 16:15:58.289227 22911 leader_election.cc:302] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b846637c584c420ab1325d5fd0bed9ae; no voters:
I20260924 16:15:58.289656 22911 leader_election.cc:288] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:58.289772 22916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:58.290011 22916 raft_consensus.cc:686] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [term 1 LEADER]: Becoming Leader. State: Replica: b846637c584c420ab1325d5fd0bed9ae, State: Running, Role: LEADER
I20260924 16:15:58.290414 22916 consensus_queue.cc:237] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b846637c584c420ab1325d5fd0bed9ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41487 } }
I20260924 16:15:58.291126 22911 sys_catalog.cc:564] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:58.292474 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b846637c584c420ab1325d5fd0bed9ae. Latest consensus state: current_term: 1 leader_uuid: "b846637c584c420ab1325d5fd0bed9ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b846637c584c420ab1325d5fd0bed9ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41487 } } }
I20260924 16:15:58.292502 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b846637c584c420ab1325d5fd0bed9ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b846637c584c420ab1325d5fd0bed9ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41487 } } }
I20260924 16:15:58.292663 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:15:58.292706 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:58.292987 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:58.293335 22925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:58.297724 22925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:58.304713 22925 catalog_manager.cc:1347] Generated new cluster ID: 11027acab00645b0b0a9d1d714fd945d
I20260924 16:15:58.304844 22925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:58.305049 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:58.295047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57525 (local address 127.0.0.1:41487)
0924 16:15:58.295165 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:58.295170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:58.295295 (+ 125us) server_negotiation.cc:407] Connection header received
0924 16:15:58.295404 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:58.295407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:58.295458 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:58.295555 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:58.296367 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:58.297185 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:58.298355 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:58.298737 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:58.298856 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:58.300632 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:58.300636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:58.300640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:58.300668 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:58.303106 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:58.303589 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:58.303593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:58.303595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:58.303683 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:58.303996 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:58.304000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:58.304002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:58.304261 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:58.304420 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:58.304703 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 16:15:58.304836 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 16:15:58.312877 22925 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:58.314127 22925 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:58.323647 22925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b846637c584c420ab1325d5fd0bed9ae: Generated new TSK 0
I20260924 16:15:58.324456 22925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:15:58.329809 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 22839
2026-09-24T16:15:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:58 dist-test-slave-6wvm krb5kdc[22942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:58 dist-test-slave-6wvm krb5kdc[22942](info): set up 2 sockets
Sep 24 16:15:58 dist-test-slave-6wvm krb5kdc[22942](info): commencing operation
krb5kdc: starting...
W20260924 16:15:59.392217 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:15:59 dist-test-slave-6wvm krb5kdc[22942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:15:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:15:59.515550 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:59.630180 22956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:59.630475 22956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:59.630518 22956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:59.634572 22956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:59.634650 22956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:59.634670 22956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:59.634686 22956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:59.634701 22956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:59.640040 22956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:15:59.641193 22956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:59.642560 22956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:59.651554 22962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:59.651563 22964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:59.651612 22961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:59.651616 22956 server_base.cc:1034] running on GCE node
I20260924 16:15:59.652356 22956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:59.653379 22956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:59.654567 22956 hybrid_clock.cc:648] HybridClock initialized: now 1790266559654547 us; error 29 us; skew 500 ppm
Sep 24 16:15:59 dist-test-slave-6wvm krb5kdc[22942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:59.669544 22956 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:15:59.670696 22956 webserver.cc:460] Webserver started at http://127.0.0.1:35647/ using document root <none> and password file <none>
I20260924 16:15:59.671285 22956 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:59.671350 22956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:59.671531 22956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:59.673723 22956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "d3d1bceffb2f4b76b1044dca5241dedf"
format_stamp: "Formatted at 2026-09-24 16:15:59 on dist-test-slave-6wvm"
server_key: "018887e8bb8a862f41fa7ecc7e6ee81a"
server_key_iv: "c936ebf34b17266198e7962d4d88eb2b"
server_key_version: "encryptionkey@0"
I20260924 16:15:59.674346 22956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "d3d1bceffb2f4b76b1044dca5241dedf"
format_stamp: "Formatted at 2026-09-24 16:15:59 on dist-test-slave-6wvm"
server_key: "018887e8bb8a862f41fa7ecc7e6ee81a"
server_key_iv: "c936ebf34b17266198e7962d4d88eb2b"
server_key_version: "encryptionkey@0"
I20260924 16:15:59.678217 22956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:15:59.680753 22971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:59.682078 22956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:15:59.682212 22956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "d3d1bceffb2f4b76b1044dca5241dedf"
format_stamp: "Formatted at 2026-09-24 16:15:59 on dist-test-slave-6wvm"
server_key: "018887e8bb8a862f41fa7ecc7e6ee81a"
server_key_iv: "c936ebf34b17266198e7962d4d88eb2b"
server_key_version: "encryptionkey@0"
I20260924 16:15:59.682396 22956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:59.701363 22956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:59.704644 22956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:59.704823 22956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:59.712496 22956 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45323
I20260924 16:15:59.712527 23024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45323 every 8 connection(s)
I20260924 16:15:59.713557 22956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:59.716522 23025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:59.722230 23025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf: Bootstrap starting.
I20260924 16:15:59.722780 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 22956
I20260924 16:15:59.722975 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:15:59.723263 2037 external_mini_cluster.cc:1428] Setting key 2ba2adc291a0ac056bd054e65444c230
I20260924 16:15:59.725244 23025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:59.726002 23025 log.cc:824] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:59.727998 23025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf: No bootstrap required, opened a new log
I20260924 16:15:59.731803 23025 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d1bceffb2f4b76b1044dca5241dedf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45323 } }
I20260924 16:15:59.732074 23025 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:59.732165 23025 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d1bceffb2f4b76b1044dca5241dedf, State: Initialized, Role: FOLLOWER
I20260924 16:15:59.732606 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:59.724693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41425 (local address 127.0.0.1:45323)
0924 16:15:59.725185 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:59.725204 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:59.725242 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:15:59.725955 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:59.725974 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:59.726454 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:59.726795 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:59.727233 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:59.728257 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:59.729169 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:59.729641 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:59.729816 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:59.730766 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:59.730777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:59.730790 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:59.731200 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:59.731257 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:59.731281 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:59.731396 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:59.731526 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 16:15:59.731752 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":141,"threads_started":1}
I20260924 16:15:59.732669 23025 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d1bceffb2f4b76b1044dca5241dedf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45323 } }
I20260924 16:15:59.732872 23025 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:59.732946 23025 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:59.733039 23025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:59.734086 23025 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d1bceffb2f4b76b1044dca5241dedf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45323 } }
I20260924 16:15:59.734493 23025 leader_election.cc:302] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3d1bceffb2f4b76b1044dca5241dedf; no voters:
I20260924 16:15:59.734884 23025 leader_election.cc:288] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:59.734929 23030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:59.735127 23030 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [term 1 LEADER]: Becoming Leader. State: Replica: d3d1bceffb2f4b76b1044dca5241dedf, State: Running, Role: LEADER
I20260924 16:15:59.735489 23030 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d1bceffb2f4b76b1044dca5241dedf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45323 } }
I20260924 16:15:59.735687 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:15:59.736196 23025 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:59.737430 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3d1bceffb2f4b76b1044dca5241dedf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d1bceffb2f4b76b1044dca5241dedf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45323 } } }
I20260924 16:15:59.737767 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:59.737437 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3d1bceffb2f4b76b1044dca5241dedf. Latest consensus state: current_term: 1 leader_uuid: "d3d1bceffb2f4b76b1044dca5241dedf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d1bceffb2f4b76b1044dca5241dedf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45323 } } }
I20260924 16:15:59.738104 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:59.738456 23044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:59.741513 23044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:59.742934 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:59.738057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38149 (local address 127.0.0.1:45323)
0924 16:15:59.738171 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:59.738175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:59.738489 (+ 314us) server_negotiation.cc:407] Connection header received
0924 16:15:59.738682 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:59.738685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:59.738730 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:59.738778 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:59.739208 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:59.739927 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:59.741095 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:59.741418 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:59.741520 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:59.742472 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:59.742475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:59.742478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:15:59.742491 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:59.742520 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:15:59.742537 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:59.742620 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:59.742749 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 16:15:59.742802 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:15:59.747251 23044 catalog_manager.cc:1347] Generated new cluster ID: 6d77024794d44df18014083c706d76ef
I20260924 16:15:59.747313 23044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:59.763363 23044 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:59.764467 23044 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:59.772483 23044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3d1bceffb2f4b76b1044dca5241dedf: Generated new TSK 0
I20260924 16:15:59.773407 23044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:15:59.783200 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 22956
2026-09-24T16:15:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:59 dist-test-slave-6wvm krb5kdc[23057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:59 dist-test-slave-6wvm krb5kdc[23057](info): set up 2 sockets
Sep 24 16:15:59 dist-test-slave-6wvm krb5kdc[23057](info): commencing operation
krb5kdc: starting...
W20260924 16:16:00.826402 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:00 dist-test-slave-6wvm krb5kdc[23057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:16:00.949283 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:01.064205 23070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:01.064531 23070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:01.064612 23070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:01.068804 23070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:01.068913 23070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:01.068961 23070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:01.069000 23070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:01.069029 23070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:01.074735 23070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:01.075984 23070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:01.077137 23070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:01.086040 23078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:01.086040 23075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:01.086162 23070 server_base.cc:1034] running on GCE node
W20260924 16:16:01.086042 23076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:01.086869 23070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:01.088069 23070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:01.089272 23070 hybrid_clock.cc:648] HybridClock initialized: now 1790266561089251 us; error 33 us; skew 500 ppm
Sep 24 16:16:01 dist-test-slave-6wvm krb5kdc[23057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:01.092162 23070 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:16:01.093029 23070 webserver.cc:460] Webserver started at http://127.0.0.1:37145/ using document root <none> and password file <none>
I20260924 16:16:01.093560 23070 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:01.093623 23070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:01.093791 23070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:01.095906 23070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "76f4842f09024471a5bed53df29bbe62"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-6wvm"
server_key: "3844afd70a03d675f592592c93aaf7a8"
server_key_iv: "f41548b939007e1b2d3e60ad968dc711"
server_key_version: "encryptionkey@0"
I20260924 16:16:01.096438 23070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "76f4842f09024471a5bed53df29bbe62"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-6wvm"
server_key: "3844afd70a03d675f592592c93aaf7a8"
server_key_iv: "f41548b939007e1b2d3e60ad968dc711"
server_key_version: "encryptionkey@0"
I20260924 16:16:01.100256 23070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:16:01.102802 23085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:01.104147 23070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:01.104280 23070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "76f4842f09024471a5bed53df29bbe62"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-6wvm"
server_key: "3844afd70a03d675f592592c93aaf7a8"
server_key_iv: "f41548b939007e1b2d3e60ad968dc711"
server_key_version: "encryptionkey@0"
I20260924 16:16:01.104394 23070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:01.109252 23070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:01.112706 23070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:01.112921 23070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:01.120325 23070 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38255
I20260924 16:16:01.120357 23138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38255 every 8 connection(s)
I20260924 16:16:01.121327 23070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:01.124332 23139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:01.126034 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 23070
I20260924 16:16:01.126212 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:01.126506 2037 external_mini_cluster.cc:1428] Setting key 126e85fd2029fc5fdfb87306b980dd82
I20260924 16:16:01.130801 23139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62: Bootstrap starting.
I20260924 16:16:01.131749 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:01.127940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47249 (local address 127.0.0.1:38255)
0924 16:16:01.128378 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:01.128394 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:01.128429 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:16:01.129082 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:01.129100 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:01.129370 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:01.129709 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:01.130546 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:01.130551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:01.130563 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:01.130573 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:01.130634 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:01.130654 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:01.130807 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:01.130878 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 16:16:01.131040 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":137,"threads_started":1}
I20260924 16:16:01.133248 23139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:01.134140 23139 log.cc:824] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:01.134680 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:01.136555 23139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62: No bootstrap required, opened a new log
I20260924 16:16:01.138619 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:01.136995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40691 (local address 127.0.0.1:38255)
0924 16:16:01.137119 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:01.137124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:01.137275 (+ 151us) server_negotiation.cc:407] Connection header received
0924 16:16:01.137371 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:01.137374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:01.137409 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:01.137496 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:01.138213 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:01.138219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:01.138222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:01.138255 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:01.138283 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:01.138297 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:01.138384 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:01.138453 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 16:16:01.138492 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:16:01.140609 23139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f4842f09024471a5bed53df29bbe62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38255 } }
I20260924 16:16:01.140887 23139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:01.140959 23139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f4842f09024471a5bed53df29bbe62, State: Initialized, Role: FOLLOWER
I20260924 16:16:01.141463 23139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f4842f09024471a5bed53df29bbe62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38255 } }
I20260924 16:16:01.141592 23139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:01.141671 23139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:01.141778 23139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:01.142828 23139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f4842f09024471a5bed53df29bbe62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38255 } }
I20260924 16:16:01.143203 23139 leader_election.cc:302] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f4842f09024471a5bed53df29bbe62; no voters:
I20260924 16:16:01.143534 23139 leader_election.cc:288] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:01.143658 23150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:01.143893 23150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [term 1 LEADER]: Becoming Leader. State: Replica: 76f4842f09024471a5bed53df29bbe62, State: Running, Role: LEADER
I20260924 16:16:01.144306 23150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f4842f09024471a5bed53df29bbe62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38255 } }
I20260924 16:16:01.144935 23139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:01.146147 23152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f4842f09024471a5bed53df29bbe62. Latest consensus state: current_term: 1 leader_uuid: "76f4842f09024471a5bed53df29bbe62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f4842f09024471a5bed53df29bbe62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38255 } } }
I20260924 16:16:01.146312 23152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:01.146695 23151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76f4842f09024471a5bed53df29bbe62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f4842f09024471a5bed53df29bbe62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38255 } } }
I20260924 16:16:01.146811 23151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62 [sys.catalog]: This master's current role is: LEADER
W20260924 16:16:01.150118 23165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:16:01.150213 23165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:16:01.150341 23159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:01.151289 23159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:01.157086 23159 catalog_manager.cc:1347] Generated new cluster ID: be312a9159b94192a7ea564a69107f38
I20260924 16:16:01.157166 23159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:01.169696 23159 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:01.170089 23159 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:01.174468 23159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76f4842f09024471a5bed53df29bbe62: Generated new TSK 0
I20260924 16:16:01.175195 23159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:01.179677 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 23070
2026-09-24T16:16:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:01 dist-test-slave-6wvm krb5kdc[23173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:01 dist-test-slave-6wvm krb5kdc[23173](info): set up 2 sockets
Sep 24 16:16:01 dist-test-slave-6wvm krb5kdc[23173](info): commencing operation
krb5kdc: starting...
W20260924 16:16:02.283476 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:02 dist-test-slave-6wvm krb5kdc[23173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:16:02.412801 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:02.526196 23186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:02.526476 23186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:02.526556 23186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:02.530524 23186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:02.530580 23186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:02.530599 23186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:02.530661 23186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:02.530690 23186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:02.535928 23186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:02.537042 23186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:02.538101 23186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:02.547011 23191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:02.547021 23192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:02.547055 23194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:02.547363 23186 server_base.cc:1034] running on GCE node
I20260924 16:16:02.547994 23186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:02.549192 23186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:02.550380 23186 hybrid_clock.cc:648] HybridClock initialized: now 1790266562550345 us; error 45 us; skew 500 ppm
Sep 24 16:16:02 dist-test-slave-6wvm krb5kdc[23173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:02.553030 23186 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:16:02.553953 23186 webserver.cc:460] Webserver started at http://127.0.0.1:43795/ using document root <none> and password file <none>
I20260924 16:16:02.554571 23186 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:02.554658 23186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:02.554883 23186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:02.556911 23186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "33a108fb85ac436e81bfe5ba33274a72"
format_stamp: "Formatted at 2026-09-24 16:16:02 on dist-test-slave-6wvm"
server_key: "8750b6e809a8f022855c9da76076ac6d"
server_key_iv: "8fcf69debd0dfdbc64c06735cda43274"
server_key_version: "encryptionkey@0"
I20260924 16:16:02.557478 23186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "33a108fb85ac436e81bfe5ba33274a72"
format_stamp: "Formatted at 2026-09-24 16:16:02 on dist-test-slave-6wvm"
server_key: "8750b6e809a8f022855c9da76076ac6d"
server_key_iv: "8fcf69debd0dfdbc64c06735cda43274"
server_key_version: "encryptionkey@0"
I20260924 16:16:02.561306 23186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 16:16:02.563802 23201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:02.564981 23186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:16:02.565121 23186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "33a108fb85ac436e81bfe5ba33274a72"
format_stamp: "Formatted at 2026-09-24 16:16:02 on dist-test-slave-6wvm"
server_key: "8750b6e809a8f022855c9da76076ac6d"
server_key_iv: "8fcf69debd0dfdbc64c06735cda43274"
server_key_version: "encryptionkey@0"
I20260924 16:16:02.565232 23186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:02.578946 23186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:02.581821 23186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:02.582053 23186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:02.589956 23186 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40465
I20260924 16:16:02.589972 23253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40465 every 8 connection(s)
I20260924 16:16:02.591025 23186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:02.593880 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:02.599149 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 23186
I20260924 16:16:02.599308 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:02.599593 2037 external_mini_cluster.cc:1428] Setting key ad7a9cc22382da08af76b78d4a5c8647
I20260924 16:16:02.599812 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72: Bootstrap starting.
I20260924 16:16:02.602624 23254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:02.603401 23254 log.cc:824] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:02.605360 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72: No bootstrap required, opened a new log
Sep 24 16:16:02 dist-test-slave-6wvm krb5kdc[23173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:16:02.609530 23254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a108fb85ac436e81bfe5ba33274a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } }
I20260924 16:16:02.609776 23254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:02.609856 23254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a108fb85ac436e81bfe5ba33274a72, State: Initialized, Role: FOLLOWER
I20260924 16:16:02.610385 23254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a108fb85ac436e81bfe5ba33274a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } }
I20260924 16:16:02.610563 23254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:02.610652 23254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:02.610775 23254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:02.611699 23254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a108fb85ac436e81bfe5ba33274a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } }
I20260924 16:16:02.612046 23254 leader_election.cc:302] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33a108fb85ac436e81bfe5ba33274a72; no voters:
I20260924 16:16:02.612449 23254 leader_election.cc:288] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:02.612691 23260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:02.612969 23260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [term 1 LEADER]: Becoming Leader. State: Replica: 33a108fb85ac436e81bfe5ba33274a72, State: Running, Role: LEADER
I20260924 16:16:02.613279 23260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a108fb85ac436e81bfe5ba33274a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } }
I20260924 16:16:02.613862 23254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:02.614559 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33a108fb85ac436e81bfe5ba33274a72. Latest consensus state: current_term: 1 leader_uuid: "33a108fb85ac436e81bfe5ba33274a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a108fb85ac436e81bfe5ba33274a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } } }
I20260924 16:16:02.614710 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:02.615085 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33a108fb85ac436e81bfe5ba33274a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a108fb85ac436e81bfe5ba33274a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } } }
I20260924 16:16:02.615235 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:02.615463 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.601009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:59353 (local address 127.0.0.1:40465)
0924 16:16:02.601511 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.601526 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.601564 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:16:02.602360 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.602383 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.602784 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.603156 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:02.603999 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.604993 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.605923 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.606458 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.606634 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.609452 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:02.609457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:02.609472 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:02.609500 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:02.611478 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.612006 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.612012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.612018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.612151 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.612495 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.612498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.612500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.613031 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:02.613827 (+ 796us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.614050 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.614308 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":145,"threads_started":1}
I20260924 16:16:02.618814 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 16:16:02.619234 23275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:16:02.619311 23275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:16:02.619386 23267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:02.620280 23267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:02.626935 23267 catalog_manager.cc:1347] Generated new cluster ID: f4cca93d59ef433da4f47d5708d9cb60
I20260924 16:16:02.627295 23267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:02.629921 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.620867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:39463 (local address 127.0.0.1:40465)
0924 16:16:02.621001 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.621005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.621181 (+ 176us) server_negotiation.cc:407] Connection header received
0924 16:16:02.621313 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.621316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.621359 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.621445 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:02.622108 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.622690 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.623522 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.623788 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.623893 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.625992 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:02.625997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:02.626000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:02.626028 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:02.627973 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.628539 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.628543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.628545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.628637 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.628973 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.628976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.628978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.629204 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:02.629355 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.629628 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.629745 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":44}
I20260924 16:16:02.633741 23267 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:02.634995 23267 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:02.647423 23267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33a108fb85ac436e81bfe5ba33274a72: Generated new TSK 0
I20260924 16:16:02.648301 23267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:02.653621 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 23186
W20260924 16:16:02.660164 23170 connection.cc:535] client connection to 127.0.0.1:40465 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40465): Connection reset by peer (error 104)
2026-09-24T16:16:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:02 dist-test-slave-6wvm krb5kdc[23286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:02 dist-test-slave-6wvm krb5kdc[23286](info): set up 2 sockets
Sep 24 16:16:02 dist-test-slave-6wvm krb5kdc[23286](info): commencing operation
krb5kdc: starting...
W20260924 16:16:03.746161 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:03 dist-test-slave-6wvm krb5kdc[23286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:16:03.873719 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:03.988152 23299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:03.988428 23299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:03.988471 23299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:03.992439 23299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:03.992492 23299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:03.992511 23299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:03.992527 23299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:03.992542 23299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:03.997619 23299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:03.998641 23299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:03.999732 23299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:04.008538 23305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:04.008500 23304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:04.008584 23307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:04.008711 23299 server_base.cc:1034] running on GCE node
I20260924 16:16:04.009451 23299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:04.010699 23299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:04.011885 23299 hybrid_clock.cc:648] HybridClock initialized: now 1790266564011853 us; error 50 us; skew 500 ppm
Sep 24 16:16:04 dist-test-slave-6wvm krb5kdc[23286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:04.014696 23299 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:16:04.015587 23299 webserver.cc:460] Webserver started at http://127.0.0.1:36415/ using document root <none> and password file <none>
I20260924 16:16:04.016140 23299 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:04.016198 23299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:04.016359 23299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:04.018476 23299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "4d225740d5c04c0fb4ad288a55e9f141"
format_stamp: "Formatted at 2026-09-24 16:16:04 on dist-test-slave-6wvm"
server_key: "70d6e593bae505452916637d86bd6930"
server_key_iv: "52c62a1cef56c2494570f35d153780b4"
server_key_version: "encryptionkey@0"
I20260924 16:16:04.018994 23299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "4d225740d5c04c0fb4ad288a55e9f141"
format_stamp: "Formatted at 2026-09-24 16:16:04 on dist-test-slave-6wvm"
server_key: "70d6e593bae505452916637d86bd6930"
server_key_iv: "52c62a1cef56c2494570f35d153780b4"
server_key_version: "encryptionkey@0"
I20260924 16:16:04.022631 23299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:04.024883 23314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:04.026127 23299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:16:04.026226 23299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "4d225740d5c04c0fb4ad288a55e9f141"
format_stamp: "Formatted at 2026-09-24 16:16:04 on dist-test-slave-6wvm"
server_key: "70d6e593bae505452916637d86bd6930"
server_key_iv: "52c62a1cef56c2494570f35d153780b4"
server_key_version: "encryptionkey@0"
I20260924 16:16:04.026379 23299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:04.050660 23299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:04.053815 23299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:04.053978 23299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:04.061182 23299 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41669
I20260924 16:16:04.061209 23368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41669 every 8 connection(s)
I20260924 16:16:04.062275 23299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:04.065173 23369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:04.070724 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 23299
I20260924 16:16:04.070875 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:04.071098 23369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141: Bootstrap starting.
I20260924 16:16:04.071197 2037 external_mini_cluster.cc:1428] Setting key 5afccfb990cf2f6f033c4957ac97431a
I20260924 16:16:04.073741 23369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:04.074571 23369 log.cc:824] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:04.076583 23369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141: No bootstrap required, opened a new log
I20260924 16:16:04.080440 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:04.072654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:42655 (local address 127.0.0.1:41669)
0924 16:16:04.073083 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:04.073099 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:04.073139 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:16:04.073812 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:04.073839 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:04.074215 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:04.074615 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:04.075065 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:04.076164 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:04.077108 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:04.077580 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:04.077789 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:04.078723 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:04.078728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:04.078741 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:04.079137 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:04.079190 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:04.079211 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:04.079366 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:04.079547 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 16:16:04.079697 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":131,"threads_started":1}
I20260924 16:16:04.080381 23369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d225740d5c04c0fb4ad288a55e9f141" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41669 } }
I20260924 16:16:04.080724 23369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:04.080792 23369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d225740d5c04c0fb4ad288a55e9f141, State: Initialized, Role: FOLLOWER
I20260924 16:16:04.081272 23369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d225740d5c04c0fb4ad288a55e9f141" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41669 } }
I20260924 16:16:04.081451 23369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:04.081522 23369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:04.081640 23369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:04.082943 23369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d225740d5c04c0fb4ad288a55e9f141" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41669 } }
I20260924 16:16:04.083355 23369 leader_election.cc:302] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d225740d5c04c0fb4ad288a55e9f141; no voters:
I20260924 16:16:04.083712 23369 leader_election.cc:288] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:04.084002 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:04.084040 23374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:04.084223 23374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [term 1 LEADER]: Becoming Leader. State: Replica: 4d225740d5c04c0fb4ad288a55e9f141, State: Running, Role: LEADER
I20260924 16:16:04.084578 23374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d225740d5c04c0fb4ad288a55e9f141" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41669 } }
I20260924 16:16:04.085070 23369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:04.085829 23374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d225740d5c04c0fb4ad288a55e9f141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d225740d5c04c0fb4ad288a55e9f141" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41669 } } }
I20260924 16:16:04.086056 23374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:04.086525 23384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:04.086484 23375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d225740d5c04c0fb4ad288a55e9f141. Latest consensus state: current_term: 1 leader_uuid: "4d225740d5c04c0fb4ad288a55e9f141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d225740d5c04c0fb4ad288a55e9f141" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41669 } } }
I20260924 16:16:04.086772 23375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:04.090147 23384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:04.095438 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:04.089887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:39523 (local address 127.0.0.1:41669)
0924 16:16:04.089992 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:04.089996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:04.090637 (+ 641us) server_negotiation.cc:407] Connection header received
0924 16:16:04.090842 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:04.090846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:04.090900 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:04.090951 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:04.091428 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:04.092230 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:04.093465 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:04.093783 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:04.093897 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:04.094932 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:04.094935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:04.094938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:04.094952 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:04.094979 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:04.094991 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:04.095076 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:04.095270 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 16:16:04.095304 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 16:16:04.096618 23384 catalog_manager.cc:1347] Generated new cluster ID: bf6e6113167444ba9bc798e66188034f
I20260924 16:16:04.096681 23384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:04.121440 23384 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:04.122669 23384 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:04.130033 23384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d225740d5c04c0fb4ad288a55e9f141: Generated new TSK 0
I20260924 16:16:04.130996 23384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:04.144677 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 23299
W20260924 16:16:04.151437 23284 connection.cc:535] client connection to 127.0.0.1:41669 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41669): Connection reset by peer (error 104)
2026-09-24T16:16:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1491 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:04 dist-test-slave-6wvm krb5kdc[23400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:04 dist-test-slave-6wvm krb5kdc[23400](info): set up 2 sockets
Sep 24 16:16:04 dist-test-slave-6wvm krb5kdc[23400](info): commencing operation
krb5kdc: starting...
W20260924 16:16:05.182140 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:05 dist-test-slave-6wvm krb5kdc[23400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:16:05.305711 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:05.422009 23414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:05.422374 23414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:05.422474 23414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:05.426537 23414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:05.426668 23414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:05.426705 23414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:05.426756 23414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:05.426795 23414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:05.432453 23414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:05.433681 23414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:05.434855 23414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:16:05.444285 23414 server_base.cc:1034] running on GCE node
W20260924 16:16:05.444211 23421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:05.444227 23420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:05.444239 23423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:05.445210 23414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:05.446574 23414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:05.447768 23414 hybrid_clock.cc:648] HybridClock initialized: now 1790266565447754 us; error 30 us; skew 500 ppm
Sep 24 16:16:05 dist-test-slave-6wvm krb5kdc[23400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:05.453440 23414 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:16:05.454510 23414 webserver.cc:460] Webserver started at http://127.0.0.1:36559/ using document root <none> and password file <none>
I20260924 16:16:05.455114 23414 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:05.455202 23414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:05.455505 23414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:05.457614 23414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "094569b6619c4f8fa0df83d35e813b20"
format_stamp: "Formatted at 2026-09-24 16:16:05 on dist-test-slave-6wvm"
server_key: "5889f6abfb2afde22933f56ece050148"
server_key_iv: "f87188b3125d4c59c1c215736ba3be95"
server_key_version: "encryptionkey@0"
I20260924 16:16:05.458201 23414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "094569b6619c4f8fa0df83d35e813b20"
format_stamp: "Formatted at 2026-09-24 16:16:05 on dist-test-slave-6wvm"
server_key: "5889f6abfb2afde22933f56ece050148"
server_key_iv: "f87188b3125d4c59c1c215736ba3be95"
server_key_version: "encryptionkey@0"
I20260924 16:16:05.462023 23414 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:05.464695 23430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:05.466024 23414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:16:05.466171 23414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "094569b6619c4f8fa0df83d35e813b20"
format_stamp: "Formatted at 2026-09-24 16:16:05 on dist-test-slave-6wvm"
server_key: "5889f6abfb2afde22933f56ece050148"
server_key_iv: "f87188b3125d4c59c1c215736ba3be95"
server_key_version: "encryptionkey@0"
I20260924 16:16:05.466344 23414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:05.471055 23414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:05.474354 23414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:05.474545 23414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:05.481683 23414 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37991
I20260924 16:16:05.481703 23483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37991 every 8 connection(s)
I20260924 16:16:05.482699 23414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:05.485596 23484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:05.491248 23484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20: Bootstrap starting.
I20260924 16:16:05.492779 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 23414
I20260924 16:16:05.492928 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:05.493222 2037 external_mini_cluster.cc:1428] Setting key 72a3dc81d100d7c80319df44e42f2b62
I20260924 16:16:05.494024 23484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:05.494853 23484 log.cc:824] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:05.497025 23484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20: No bootstrap required, opened a new log
I20260924 16:16:05.498667 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:05.494395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:37971 (local address 127.0.0.1:37991)
0924 16:16:05.494966 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:05.494990 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:05.495032 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:16:05.495690 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:05.495707 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:05.495989 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:05.496282 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:05.497213 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:05.497218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:05.497230 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:05.497242 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:05.497297 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:05.497317 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:05.497408 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:05.497602 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 16:16:05.497831 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":195,"threads_started":1}
I20260924 16:16:05.501210 23484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094569b6619c4f8fa0df83d35e813b20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37991 } }
I20260924 16:16:05.501490 23484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:05.501554 23484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094569b6619c4f8fa0df83d35e813b20, State: Initialized, Role: FOLLOWER
I20260924 16:16:05.501799 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:05.502070 23484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094569b6619c4f8fa0df83d35e813b20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37991 } }
I20260924 16:16:05.502219 23484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:05.502306 23484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:05.502406 23484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:05.503417 23484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094569b6619c4f8fa0df83d35e813b20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37991 } }
I20260924 16:16:05.503785 23484 leader_election.cc:302] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 094569b6619c4f8fa0df83d35e813b20; no voters:
I20260924 16:16:05.504149 23484 leader_election.cc:288] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:05.504498 23496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:05.504699 23496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [term 1 LEADER]: Becoming Leader. State: Replica: 094569b6619c4f8fa0df83d35e813b20, State: Running, Role: LEADER
I20260924 16:16:05.505002 23496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094569b6619c4f8fa0df83d35e813b20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37991 } }
I20260924 16:16:05.505489 23484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:05.506126 23496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094569b6619c4f8fa0df83d35e813b20. Latest consensus state: current_term: 1 leader_uuid: "094569b6619c4f8fa0df83d35e813b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094569b6619c4f8fa0df83d35e813b20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37991 } } }
I20260924 16:16:05.506335 23496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:05.506626 23497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "094569b6619c4f8fa0df83d35e813b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094569b6619c4f8fa0df83d35e813b20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37991 } } }
I20260924 16:16:05.506784 23497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:05.506839 23504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:05.509701 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:05.504063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:48837 (local address 127.0.0.1:37991)
0924 16:16:05.504162 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:05.504165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:05.507948 (+ 3783us) server_negotiation.cc:407] Connection header received
0924 16:16:05.508152 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:05.508155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:05.508200 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:05.508299 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:05.509248 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:05.509252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:05.509255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:05.509269 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:05.509301 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:05.509315 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:05.509414 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:05.509519 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 16:16:05.509566 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 16:16:05.510620 23504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:05.516175 23504 catalog_manager.cc:1347] Generated new cluster ID: a0b9e5cdcfc64398b879e30de9e15cf0
I20260924 16:16:05.516249 23504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:05.536041 23504 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:05.536409 23504 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:05.541500 23504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 094569b6619c4f8fa0df83d35e813b20: Generated new TSK 0
I20260924 16:16:05.542204 23504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:05.546480 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 23414
2026-09-24T16:16:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:05 dist-test-slave-6wvm krb5kdc[23517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:05 dist-test-slave-6wvm krb5kdc[23517](info): set up 2 sockets
Sep 24 16:16:05 dist-test-slave-6wvm krb5kdc[23517](info): commencing operation
krb5kdc: starting...
W20260924 16:16:06.594082 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:06 dist-test-slave-6wvm krb5kdc[23517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:16:06.717429 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:06.828038 23530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:06.828344 23530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:06.828421 23530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:06.832368 23530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:06.832455 23530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:06.832494 23530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:06.832547 23530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:06.832587 23530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:06.837675 23530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34385
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:06.838745 23530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:06.839775 23530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:06.848109 23538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:06.848119 23535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:06.848250 23530 server_base.cc:1034] running on GCE node
W20260924 16:16:06.848407 23536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:06.848976 23530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:06.850016 23530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:06.851202 23530 hybrid_clock.cc:648] HybridClock initialized: now 1790266566851170 us; error 52 us; skew 500 ppm
Sep 24 16:16:06 dist-test-slave-6wvm krb5kdc[23517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:06.853878 23530 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:16:06.854769 23530 webserver.cc:460] Webserver started at http://127.0.0.1:36201/ using document root <none> and password file <none>
I20260924 16:16:06.855291 23530 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:06.855345 23530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:06.855525 23530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:06.857558 23530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "1bb27d46740f49f2a5c555a61a25f1c1"
format_stamp: "Formatted at 2026-09-24 16:16:06 on dist-test-slave-6wvm"
server_key: "858503de3b3fc8819eda7b7a32bdc531"
server_key_iv: "96081bdcfb470527a575d0eb1a902175"
server_key_version: "encryptionkey@0"
I20260924 16:16:06.858088 23530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "1bb27d46740f49f2a5c555a61a25f1c1"
format_stamp: "Formatted at 2026-09-24 16:16:06 on dist-test-slave-6wvm"
server_key: "858503de3b3fc8819eda7b7a32bdc531"
server_key_iv: "96081bdcfb470527a575d0eb1a902175"
server_key_version: "encryptionkey@0"
I20260924 16:16:06.861747 23530 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:06.864300 23545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:06.865568 23530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:16:06.865675 23530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "1bb27d46740f49f2a5c555a61a25f1c1"
format_stamp: "Formatted at 2026-09-24 16:16:06 on dist-test-slave-6wvm"
server_key: "858503de3b3fc8819eda7b7a32bdc531"
server_key_iv: "96081bdcfb470527a575d0eb1a902175"
server_key_version: "encryptionkey@0"
I20260924 16:16:06.865765 23530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:06.891008 23530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:06.894541 23530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:06.894714 23530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:06.902092 23530 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34385
I20260924 16:16:06.902105 23597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34385 every 8 connection(s)
I20260924 16:16:06.903111 23530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:06.903514 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 23530
I20260924 16:16:06.903671 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:06.903959 2037 external_mini_cluster.cc:1428] Setting key afaf29f41115e2abb4f051501897ef1b
I20260924 16:16:06.906100 23598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:06.912971 23598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1: Bootstrap starting.
Sep 24 16:16:06 dist-test-slave-6wvm krb5kdc[23517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:16:06.915268 23598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:06.915990 23598 log.cc:824] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:06.917914 23598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1: No bootstrap required, opened a new log
I20260924 16:16:06.918529 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:06.905413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:47779 (local address 127.0.0.1:34385)
0924 16:16:06.905827 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:06.905843 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:06.905879 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:16:06.906573 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:06.906591 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:06.906945 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:06.907222 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:06.908164 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:06.909154 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:06.910342 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:06.910810 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:06.910927 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:06.913773 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:06.913778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:06.913792 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:06.913820 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:06.915546 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:06.916060 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:06.916065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:06.916069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:06.916221 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:06.916552 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:06.916556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:06.916557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:06.916948 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:06.917179 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:06.917499 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 16:16:06.917727 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":137,"threads_started":1}
I20260924 16:16:06.921669 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:06.922112 23598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb27d46740f49f2a5c555a61a25f1c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34385 } }
I20260924 16:16:06.922395 23598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:06.922477 23598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb27d46740f49f2a5c555a61a25f1c1, State: Initialized, Role: FOLLOWER
I20260924 16:16:06.922904 23598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb27d46740f49f2a5c555a61a25f1c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34385 } }
I20260924 16:16:06.923098 23598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:06.923172 23598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:06.923286 23598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:06.924440 23598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb27d46740f49f2a5c555a61a25f1c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34385 } }
I20260924 16:16:06.924793 23598 leader_election.cc:302] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb27d46740f49f2a5c555a61a25f1c1; no voters:
I20260924 16:16:06.925105 23598 leader_election.cc:288] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:06.925240 23608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:06.925472 23608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb27d46740f49f2a5c555a61a25f1c1, State: Running, Role: LEADER
I20260924 16:16:06.925850 23608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb27d46740f49f2a5c555a61a25f1c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34385 } }
I20260924 16:16:06.926420 23598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:06.927536 23610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bb27d46740f49f2a5c555a61a25f1c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb27d46740f49f2a5c555a61a25f1c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34385 } } }
I20260924 16:16:06.927755 23610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:06.928038 23610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bb27d46740f49f2a5c555a61a25f1c1. Latest consensus state: current_term: 1 leader_uuid: "1bb27d46740f49f2a5c555a61a25f1c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb27d46740f49f2a5c555a61a25f1c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34385 } } }
I20260924 16:16:06.928194 23610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:06.929013 23619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:06.932168 23619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:06.933012 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:06.923846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:45855 (local address 127.0.0.1:34385)
0924 16:16:06.923969 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:06.923974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:06.924133 (+ 159us) server_negotiation.cc:407] Connection header received
0924 16:16:06.924240 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:06.924245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:06.924300 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:06.924380 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:06.925090 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:06.925870 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:06.927125 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:06.927529 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:06.927634 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:06.929454 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:06.929457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:06.929459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:06.929480 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:06.931073 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:06.931626 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:06.931630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:06.931631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:06.931722 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:06.932086 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:06.932090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:06.932091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:06.932291 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:06.932442 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:06.932761 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 16:16:06.932861 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:16:06.937667 23619 catalog_manager.cc:1347] Generated new cluster ID: 02925f8ad92a498aa3566cc7c8607977
I20260924 16:16:06.937729 23619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:06.952659 23619 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:06.954010 23619 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:06.982378 23619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bb27d46740f49f2a5c555a61a25f1c1: Generated new TSK 0
I20260924 16:16:06.983283 23619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:06.991219 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 23530
W20260924 16:16:06.997980 23514 connection.cc:535] client connection to 127.0.0.1:34385 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34385): Connection reset by peer (error 104)
2026-09-24T16:16:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:07 dist-test-slave-6wvm krb5kdc[23630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:07 dist-test-slave-6wvm krb5kdc[23630](info): set up 2 sockets
Sep 24 16:16:07 dist-test-slave-6wvm krb5kdc[23630](info): commencing operation
krb5kdc: starting...
W20260924 16:16:08.074515 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:08 dist-test-slave-6wvm krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:16:08.205559 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:08.322571 23643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:08.322883 23643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:08.322962 23643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:08.327078 23643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:08.327164 23643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:08.327203 23643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:08.327257 23643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:08.327293 23643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:08.332643 23643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:08.333817 23643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:08.334951 23643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:08.343590 23648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:08.343573 23649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:08.343586 23651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:08.343933 23643 server_base.cc:1034] running on GCE node
I20260924 16:16:08.344518 23643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:08.345584 23643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:08.346769 23643 hybrid_clock.cc:648] HybridClock initialized: now 1790266568346738 us; error 42 us; skew 500 ppm
Sep 24 16:16:08 dist-test-slave-6wvm krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:08.349453 23643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:16:08.350450 23643 webserver.cc:460] Webserver started at http://127.0.0.1:43593/ using document root <none> and password file <none>
I20260924 16:16:08.351014 23643 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:08.351102 23643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:08.351331 23643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:08.353404 23643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "8ebccb9e0f5b49d0a29ea25ff6acf080"
format_stamp: "Formatted at 2026-09-24 16:16:08 on dist-test-slave-6wvm"
server_key: "1417c49a09cc4174c135f2fafeb02de9"
server_key_iv: "d664ee91cdea8d48fbb9c91ccc18f9ab"
server_key_version: "encryptionkey@0"
I20260924 16:16:08.353993 23643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "8ebccb9e0f5b49d0a29ea25ff6acf080"
format_stamp: "Formatted at 2026-09-24 16:16:08 on dist-test-slave-6wvm"
server_key: "1417c49a09cc4174c135f2fafeb02de9"
server_key_iv: "d664ee91cdea8d48fbb9c91ccc18f9ab"
server_key_version: "encryptionkey@0"
I20260924 16:16:08.357761 23643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:16:08.360373 23658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:08.361685 23643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:16:08.361858 23643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "8ebccb9e0f5b49d0a29ea25ff6acf080"
format_stamp: "Formatted at 2026-09-24 16:16:08 on dist-test-slave-6wvm"
server_key: "1417c49a09cc4174c135f2fafeb02de9"
server_key_iv: "d664ee91cdea8d48fbb9c91ccc18f9ab"
server_key_version: "encryptionkey@0"
I20260924 16:16:08.361980 23643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:08.372064 23643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:08.375090 23643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:08.375279 23643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:08.382272 23643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41939
I20260924 16:16:08.382282 23711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41939 every 8 connection(s)
I20260924 16:16:08.383258 23643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:08.386175 23712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:08.391921 23712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080: Bootstrap starting.
I20260924 16:16:08.392671 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 23643
I20260924 16:16:08.392813 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:08.393115 2037 external_mini_cluster.cc:1428] Setting key 3e3deeb023e66b5eeb1fd8d0d49a07c3
I20260924 16:16:08.395002 23712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:08.395749 23712 log.cc:824] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:08.397832 23712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080: No bootstrap required, opened a new log
I20260924 16:16:08.400913 23712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ebccb9e0f5b49d0a29ea25ff6acf080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } }
I20260924 16:16:08.401166 23712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:08.401264 23712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebccb9e0f5b49d0a29ea25ff6acf080, State: Initialized, Role: FOLLOWER
I20260924 16:16:08.401820 23712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ebccb9e0f5b49d0a29ea25ff6acf080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } }
I20260924 16:16:08.402004 23712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:08.402083 23712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:08.402204 23712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:08.403523 23712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ebccb9e0f5b49d0a29ea25ff6acf080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } }
I20260924 16:16:08.403903 23712 leader_election.cc:302] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ebccb9e0f5b49d0a29ea25ff6acf080; no voters:
I20260924 16:16:08.404259 23712 leader_election.cc:288] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:08.404366 23717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:08.404464 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:08.394501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:41667 (local address 127.0.0.1:41939)
0924 16:16:08.395022 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:08.395040 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:08.395079 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:16:08.395841 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:08.395893 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:08.396403 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:08.396855 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:08.397337 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:08.398583 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:08.399654 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:08.400912 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:08.401125 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:08.402277 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:08.402285 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:08.402305 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:08.402843 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:08.402913 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:08.402945 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:08.403084 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:08.403312 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 16:16:08.403468 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":145,"threads_started":1}
I20260924 16:16:08.404613 23717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [term 1 LEADER]: Becoming Leader. State: Replica: 8ebccb9e0f5b49d0a29ea25ff6acf080, State: Running, Role: LEADER
I20260924 16:16:08.405095 23717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ebccb9e0f5b49d0a29ea25ff6acf080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } }
I20260924 16:16:08.405586 23712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:08.406673 23719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ebccb9e0f5b49d0a29ea25ff6acf080. Latest consensus state: current_term: 1 leader_uuid: "8ebccb9e0f5b49d0a29ea25ff6acf080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ebccb9e0f5b49d0a29ea25ff6acf080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } } }
I20260924 16:16:08.406793 23719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:08.407126 23718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ebccb9e0f5b49d0a29ea25ff6acf080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ebccb9e0f5b49d0a29ea25ff6acf080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41939 } } }
I20260924 16:16:08.407217 23718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:08.407598 23726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:08.408160 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:08.410879 23726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:08.414898 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:08.410409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:48247 (local address 127.0.0.1:41939)
0924 16:16:08.410521 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:08.410525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:08.410788 (+ 263us) server_negotiation.cc:407] Connection header received
0924 16:16:08.410938 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:08.410944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:08.410997 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:08.411085 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:08.411384 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:08.411906 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:08.412897 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:08.413253 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:08.413361 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:08.414360 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:08.414364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:08.414367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:08.414381 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:08.414412 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:08.414425 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:08.414529 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:08.414711 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 16:16:08.414745 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:16:08.417150 23726 catalog_manager.cc:1347] Generated new cluster ID: f0a87dcabfdf4aaba98c7b3936a30528
I20260924 16:16:08.417217 23726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:08.440037 23726 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:08.441447 23726 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:08.469448 23726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ebccb9e0f5b49d0a29ea25ff6acf080: Generated new TSK 0
I20260924 16:16:08.470378 23726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:08.486917 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 23643
W20260924 16:16:08.493924 23628 connection.cc:535] client connection to 127.0.0.1:41939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41939): Connection reset by peer (error 104)
2026-09-24T16:16:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:08 dist-test-slave-6wvm krb5kdc[23743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:08 dist-test-slave-6wvm krb5kdc[23743](info): set up 2 sockets
Sep 24 16:16:08 dist-test-slave-6wvm krb5kdc[23743](info): commencing operation
krb5kdc: starting...
W20260924 16:16:09.590024 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:09 dist-test-slave-6wvm krb5kdc[23743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:16:09.712707 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:09.825845 23756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:09.826112 23756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:09.826152 23756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:09.830117 23756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:09.830174 23756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:09.830193 23756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:09.830209 23756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:09.830224 23756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:09.835366 23756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:09.836362 23756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:09.837332 23756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:09.845783 23765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:09.845782 23763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:09.845835 23762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:09.846113 23756 server_base.cc:1034] running on GCE node
I20260924 16:16:09.846772 23756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:09.847826 23756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:09.848996 23756 hybrid_clock.cc:648] HybridClock initialized: now 1790266569848956 us; error 50 us; skew 500 ppm
Sep 24 16:16:09 dist-test-slave-6wvm krb5kdc[23743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:09.851661 23756 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:16:09.852586 23756 webserver.cc:460] Webserver started at http://127.0.0.1:45623/ using document root <none> and password file <none>
I20260924 16:16:09.853156 23756 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:09.853243 23756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:09.853473 23756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:09.855573 23756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "7df864b613a145c98fa9b48e6589dc06"
format_stamp: "Formatted at 2026-09-24 16:16:09 on dist-test-slave-6wvm"
server_key: "b0f3a48024251a3955046c57f4ee3e78"
server_key_iv: "dc586ec0fd6a682fd50046f383a3f0c7"
server_key_version: "encryptionkey@0"
I20260924 16:16:09.856134 23756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "7df864b613a145c98fa9b48e6589dc06"
format_stamp: "Formatted at 2026-09-24 16:16:09 on dist-test-slave-6wvm"
server_key: "b0f3a48024251a3955046c57f4ee3e78"
server_key_iv: "dc586ec0fd6a682fd50046f383a3f0c7"
server_key_version: "encryptionkey@0"
I20260924 16:16:09.859988 23756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:16:09.862530 23773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:09.863762 23756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:16:09.863907 23756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "7df864b613a145c98fa9b48e6589dc06"
format_stamp: "Formatted at 2026-09-24 16:16:09 on dist-test-slave-6wvm"
server_key: "b0f3a48024251a3955046c57f4ee3e78"
server_key_iv: "dc586ec0fd6a682fd50046f383a3f0c7"
server_key_version: "encryptionkey@0"
I20260924 16:16:09.864041 23756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:09.869208 23756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:09.872321 23756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:09.872527 23756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:09.879951 23756 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36423
I20260924 16:16:09.879963 23825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36423 every 8 connection(s)
I20260924 16:16:09.880945 23756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:09.883878 23826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:09.889011 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 23756
I20260924 16:16:09.889165 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:09.889467 2037 external_mini_cluster.cc:1428] Setting key 9ad98eaa0e0f30137f2e467ddec41452
I20260924 16:16:09.889926 23826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06: Bootstrap starting.
I20260924 16:16:09.892530 23826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:09.893397 23826 log.cc:824] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:09.895000 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:09.891015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:59659 (local address 127.0.0.1:36423)
0924 16:16:09.891482 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:09.891496 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:09.891532 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:16:09.892176 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:09.892193 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:09.892480 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:09.892861 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:09.893690 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:09.893695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:09.893708 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:09.893720 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:09.893801 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:09.893832 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:09.893950 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:09.894098 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 16:16:09.894300 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260924 16:16:09.895713 23826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06: No bootstrap required, opened a new log
I20260924 16:16:09.897846 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:09.899737 23826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df864b613a145c98fa9b48e6589dc06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36423 } }
I20260924 16:16:09.900025 23826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:09.900149 23826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df864b613a145c98fa9b48e6589dc06, State: Initialized, Role: FOLLOWER
I20260924 16:16:09.900691 23826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df864b613a145c98fa9b48e6589dc06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36423 } }
I20260924 16:16:09.900902 23826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:09.901000 23826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:09.901110 23826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:09.901786 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:09.899992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:50027 (local address 127.0.0.1:36423)
0924 16:16:09.900261 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:09.900267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:09.900287 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:16:09.900457 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:09.900459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:09.900489 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:09.900553 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:09.901389 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:09.901392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:09.901394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:09.901404 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:09.901430 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:09.901441 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:09.901509 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:09.901624 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 16:16:09.901657 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260924 16:16:09.902398 23826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df864b613a145c98fa9b48e6589dc06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36423 } }
I20260924 16:16:09.902745 23826 leader_election.cc:302] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7df864b613a145c98fa9b48e6589dc06; no voters:
I20260924 16:16:09.903059 23826 leader_election.cc:288] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:09.903192 23836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:09.903429 23836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [term 1 LEADER]: Becoming Leader. State: Replica: 7df864b613a145c98fa9b48e6589dc06, State: Running, Role: LEADER
I20260924 16:16:09.903786 23836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df864b613a145c98fa9b48e6589dc06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36423 } }
I20260924 16:16:09.904340 23826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:09.905421 23837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7df864b613a145c98fa9b48e6589dc06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df864b613a145c98fa9b48e6589dc06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36423 } } }
I20260924 16:16:09.905555 23837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:09.905881 23838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7df864b613a145c98fa9b48e6589dc06. Latest consensus state: current_term: 1 leader_uuid: "7df864b613a145c98fa9b48e6589dc06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df864b613a145c98fa9b48e6589dc06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36423 } } }
I20260924 16:16:09.906008 23845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:09.906028 23838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:09.909417 23845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:09.915162 23845 catalog_manager.cc:1347] Generated new cluster ID: 280473da26144b43885f00aad993dce7
I20260924 16:16:09.915225 23845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:09.924125 23845 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:09.924425 23845 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:09.932626 23845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7df864b613a145c98fa9b48e6589dc06: Generated new TSK 0
I20260924 16:16:09.933318 23845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:09.936700 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 23756
2026-09-24T16:16:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:09 dist-test-slave-6wvm krb5kdc[23858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:09 dist-test-slave-6wvm krb5kdc[23858](info): set up 2 sockets
Sep 24 16:16:09 dist-test-slave-6wvm krb5kdc[23858](info): commencing operation
krb5kdc: starting...
W20260924 16:16:10.978381 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:11 dist-test-slave-6wvm krb5kdc[23858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:16:11.101738 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:46653
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:46653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:11.215049 23871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:11.215306 23871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:11.215346 23871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:11.219436 23871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:11.219499 23871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:11.219519 23871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:11.219537 23871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:11.219551 23871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:11.224946 23871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:46653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:46653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:11.226066 23871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:11.227191 23871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:11.235823 23876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:11.235764 23879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:11.236037 23871 server_base.cc:1034] running on GCE node
W20260924 16:16:11.235764 23877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:11.236806 23871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:11.237893 23871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:11.239073 23871 hybrid_clock.cc:648] HybridClock initialized: now 1790266571239051 us; error 31 us; skew 500 ppm
Sep 24 16:16:11 dist-test-slave-6wvm krb5kdc[23858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266571, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:11.248476 23871 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:16:11.249428 23871 webserver.cc:460] Webserver started at http://127.1.253.126:39185/ using document root <none> and password file <none>
I20260924 16:16:11.250011 23871 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:11.250092 23871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:11.250375 23871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:11.252483 23871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "771be83651064108a661785edd5c540a"
format_stamp: "Formatted at 2026-09-24 16:16:11 on dist-test-slave-6wvm"
server_key: "ec8b9f5912c0a9a0748825901c61abb1"
server_key_iv: "58eba090fb4b1381f5bff62a72f70a17"
server_key_version: "encryptionkey@0"
I20260924 16:16:11.253090 23871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "771be83651064108a661785edd5c540a"
format_stamp: "Formatted at 2026-09-24 16:16:11 on dist-test-slave-6wvm"
server_key: "ec8b9f5912c0a9a0748825901c61abb1"
server_key_iv: "58eba090fb4b1381f5bff62a72f70a17"
server_key_version: "encryptionkey@0"
I20260924 16:16:11.257052 23871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 16:16:11.259548 23886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:11.260740 23871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:16:11.260869 23871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "771be83651064108a661785edd5c540a"
format_stamp: "Formatted at 2026-09-24 16:16:11 on dist-test-slave-6wvm"
server_key: "ec8b9f5912c0a9a0748825901c61abb1"
server_key_iv: "58eba090fb4b1381f5bff62a72f70a17"
server_key_version: "encryptionkey@0"
I20260924 16:16:11.260994 23871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:11.284152 23871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:11.287364 23871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:11.287575 23871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:11.294485 23871 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:46653
I20260924 16:16:11.294514 23941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:46653 every 8 connection(s)
I20260924 16:16:11.295434 23871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:11.298151 23942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:11.298764 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 23871
I20260924 16:16:11.298915 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:11.299238 2037 external_mini_cluster.cc:1428] Setting key c6a1b57338ea838a5ea20fba364b819b
I20260924 16:16:11.304138 23942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a: Bootstrap starting.
I20260924 16:16:11.306345 23942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:11.307063 23942 log.cc:824] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:16:11 dist-test-slave-6wvm krb5kdc[23858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:16:11.309077 23942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a: No bootstrap required, opened a new log
I20260924 16:16:11.312683 23942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771be83651064108a661785edd5c540a" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46653 } }
I20260924 16:16:11.312897 23942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:11.312952 23942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771be83651064108a661785edd5c540a, State: Initialized, Role: FOLLOWER
I20260924 16:16:11.313297 23946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:11.300702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42329 (local address 127.1.253.126:46653)
0924 16:16:11.301150 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:11.301164 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:11.301201 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:16:11.301838 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:11.301858 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:11.302194 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:11.302613 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:11.303442 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:11.304425 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:11.305270 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:11.305733 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:11.305903 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:11.308552 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:11.308556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:11.308569 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:11.308595 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:11.310345 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:11.310848 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:11.310854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:11.310858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:11.310966 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:11.311300 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:11.311304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:11.311305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:11.311695 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:11.311929 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:11.312264 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 16:16:11.312487 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":132,"threads_started":1}
I20260924 16:16:11.313453 23942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771be83651064108a661785edd5c540a" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46653 } }
I20260924 16:16:11.313612 23942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:11.313686 23942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:11.313788 23942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:11.314986 23942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771be83651064108a661785edd5c540a" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46653 } }
I20260924 16:16:11.315364 23942 leader_election.cc:302] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 771be83651064108a661785edd5c540a; no voters:
I20260924 16:16:11.315742 23942 leader_election.cc:288] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:11.315891 23948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:11.316110 23948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [term 1 LEADER]: Becoming Leader. State: Replica: 771be83651064108a661785edd5c540a, State: Running, Role: LEADER
I20260924 16:16:11.316422 23948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771be83651064108a661785edd5c540a" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46653 } }
I20260924 16:16:11.317078 23942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:11.317142 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:11.317652 23948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "771be83651064108a661785edd5c540a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771be83651064108a661785edd5c540a" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46653 } } }
I20260924 16:16:11.317757 23948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:11.317786 23949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 771be83651064108a661785edd5c540a. Latest consensus state: current_term: 1 leader_uuid: "771be83651064108a661785edd5c540a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771be83651064108a661785edd5c540a" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46653 } } }
I20260924 16:16:11.317888 23949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a [sys.catalog]: This master's current role is: LEADER
W20260924 16:16:11.322162 23967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:16:11.322260 23967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:16:11.322561 23953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:11.323593 23953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:11.327695 23946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:11.319878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55849 (local address 127.1.253.126:46653)
0924 16:16:11.319993 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:11.319996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:11.320006 (+ 10us) server_negotiation.cc:407] Connection header received
0924 16:16:11.320034 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:11.320036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:11.320069 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:11.320152 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:11.320830 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:11.321342 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:11.322176 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:11.322502 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:11.322612 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:11.324043 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:11.324047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:11.324050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:11.324080 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:11.325490 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:11.325985 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:11.325989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:11.325990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:11.326103 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:11.326561 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:11.326565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:11.326566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:11.326778 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:11.326946 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:11.327276 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 16:16:11.327547 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:16:11.330329 23953 catalog_manager.cc:1347] Generated new cluster ID: 49335d1b1da9449894ac0751a929359b
I20260924 16:16:11.330392 23953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:11.341279 23953 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:11.342162 23953 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:11.349390 23953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 771be83651064108a661785edd5c540a: Generated new TSK 0
I20260924 16:16:11.350077 23953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:11.353896 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 23871
W20260924 16:16:11.360847 23856 connection.cc:535] client connection to 127.1.253.126:46653 recv error: Network error: failed to read from TLS socket (remote: 127.1.253.126:46653): Connection reset by peer (error 104)
2026-09-24T16:16:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:11 dist-test-slave-6wvm krb5kdc[23974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:11 dist-test-slave-6wvm krb5kdc[23974](info): set up 2 sockets
Sep 24 16:16:11 dist-test-slave-6wvm krb5kdc[23974](info): commencing operation
krb5kdc: starting...
W20260924 16:16:12.399099 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:12 dist-test-slave-6wvm krb5kdc[23974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:16:12.524922 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:42045
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:45613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:42045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:12.638779 23988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:12.639084 23988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:12.639169 23988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:12.643383 23988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:12.643476 23988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:12.643515 23988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:12.643551 23988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:12.643584 23988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:12.648887 23988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:45613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:42045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:42045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:12.649986 23988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:12.651096 23988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:12.659817 23996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:12.659811 23993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:12.659950 23988 server_base.cc:1034] running on GCE node
W20260924 16:16:12.659814 23994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:12.660679 23988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:12.661768 23988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:12.662933 23988 hybrid_clock.cc:648] HybridClock initialized: now 1790266572662909 us; error 31 us; skew 500 ppm
Sep 24 16:16:12 dist-test-slave-6wvm krb5kdc[23974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266572, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:12.665597 23988 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:16:12.666591 23988 webserver.cc:460] Webserver started at http://127.1.253.126:35083/ using document root <none> and password file <none>
I20260924 16:16:12.667150 23988 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:12.667212 23988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:12.667454 23988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:12.669513 23988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "a363296e08d94bb597efb719373623f7"
format_stamp: "Formatted at 2026-09-24 16:16:12 on dist-test-slave-6wvm"
server_key: "fa3eaa8aa7db5651adbbc949aae3d793"
server_key_iv: "655bf32869e1c8d0164c6b68a8a30c17"
server_key_version: "encryptionkey@0"
I20260924 16:16:12.670089 23988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "a363296e08d94bb597efb719373623f7"
format_stamp: "Formatted at 2026-09-24 16:16:12 on dist-test-slave-6wvm"
server_key: "fa3eaa8aa7db5651adbbc949aae3d793"
server_key_iv: "655bf32869e1c8d0164c6b68a8a30c17"
server_key_version: "encryptionkey@0"
I20260924 16:16:12.673738 23988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:12.676496 24003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:12.677760 23988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:16:12.677898 23988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "a363296e08d94bb597efb719373623f7"
format_stamp: "Formatted at 2026-09-24 16:16:12 on dist-test-slave-6wvm"
server_key: "fa3eaa8aa7db5651adbbc949aae3d793"
server_key_iv: "655bf32869e1c8d0164c6b68a8a30c17"
server_key_version: "encryptionkey@0"
I20260924 16:16:12.678021 23988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:12.689288 23988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:12.692353 23988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:12.692584 23988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:12.699797 23988 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:42045
I20260924 16:16:12.699808 24057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:42045 every 8 connection(s)
I20260924 16:16:12.700824 23988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:12.701254 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 23988
I20260924 16:16:12.701419 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:12.701740 2037 external_mini_cluster.cc:1428] Setting key d01480a08df17c7b8791e36380c9fdb9
I20260924 16:16:12.704314 24058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:12.711505 24058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7: Bootstrap starting.
I20260924 16:16:12.712210 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:12.703237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39945 (local address 127.1.253.126:42045)
0924 16:16:12.703686 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:12.703705 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:12.703750 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:16:12.704625 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:12.704651 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:12.705095 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:12.705483 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:12.705910 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:12.707457 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:12.708375 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:12.709072 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:12.709237 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:12.710147 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:12.710153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:12.710171 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:12.710728 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:12.710799 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:12.710829 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:12.710965 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:12.711142 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 16:16:12.711282 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":113,"threads_started":1}
I20260924 16:16:12.713944 24058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:12.714908 24058 log.cc:824] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:12.715279 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:12.717029 24058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7: No bootstrap required, opened a new log
I20260924 16:16:12.721791 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:12.717433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50479 (local address 127.1.253.126:42045)
0924 16:16:12.717564 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:12.717568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:12.717785 (+ 217us) server_negotiation.cc:407] Connection header received
0924 16:16:12.717907 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:12.717909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:12.717947 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:12.718011 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:12.718346 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:12.718861 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:12.720055 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:12.720267 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:12.720344 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:12.721263 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:12.721266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:12.721268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:12.721281 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:12.721307 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:12.721317 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:12.721385 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:12.721572 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 16:16:12.721633 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:16:12.722007 24058 raft_consensus.cc:348] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a363296e08d94bb597efb719373623f7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42045 } }
I20260924 16:16:12.722340 24058 raft_consensus.cc:374] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:12.722402 24058 raft_consensus.cc:729] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a363296e08d94bb597efb719373623f7, State: Initialized, Role: FOLLOWER
I20260924 16:16:12.722904 24058 consensus_queue.cc:260] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a363296e08d94bb597efb719373623f7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42045 } }
I20260924 16:16:12.723065 24058 raft_consensus.cc:388] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:12.723132 24058 raft_consensus.cc:482] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:12.723230 24058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:12.724255 24058 raft_consensus.cc:504] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a363296e08d94bb597efb719373623f7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42045 } }
I20260924 16:16:12.724624 24058 leader_election.cc:302] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a363296e08d94bb597efb719373623f7; no voters:
I20260924 16:16:12.725023 24058 leader_election.cc:288] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:12.725178 24068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:12.725425 24068 raft_consensus.cc:686] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [term 1 LEADER]: Becoming Leader. State: Replica: a363296e08d94bb597efb719373623f7, State: Running, Role: LEADER
I20260924 16:16:12.725837 24068 consensus_queue.cc:237] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a363296e08d94bb597efb719373623f7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42045 } }
I20260924 16:16:12.726539 24058 sys_catalog.cc:564] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:12.727701 24070 sys_catalog.cc:455] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a363296e08d94bb597efb719373623f7. Latest consensus state: current_term: 1 leader_uuid: "a363296e08d94bb597efb719373623f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a363296e08d94bb597efb719373623f7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42045 } } }
I20260924 16:16:12.727839 24070 sys_catalog.cc:458] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:12.728134 24069 sys_catalog.cc:455] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a363296e08d94bb597efb719373623f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a363296e08d94bb597efb719373623f7" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42045 } } }
I20260924 16:16:12.728241 24069 sys_catalog.cc:458] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:12.728253 24078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:12.731216 24078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:12.737071 24078 catalog_manager.cc:1347] Generated new cluster ID: cd38d55b456641d69beb5efaad09ed30
I20260924 16:16:12.737161 24078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:12.758106 24078 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:12.759436 24078 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:12.765600 24078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a363296e08d94bb597efb719373623f7: Generated new TSK 0
I20260924 16:16:12.766489 24078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:12.778784 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 23988
W20260924 16:16:12.785346 23972 connection.cc:535] client connection to 127.1.253.126:42045 recv error: Network error: failed to read from TLS socket (remote: 127.1.253.126:42045): Connection reset by peer (error 104)
2026-09-24T16:16:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:12 dist-test-slave-6wvm krb5kdc[24090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:12 dist-test-slave-6wvm krb5kdc[24090](info): set up 2 sockets
Sep 24 16:16:12 dist-test-slave-6wvm krb5kdc[24090](info): commencing operation
krb5kdc: starting...
W20260924 16:16:13.822129 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:13 dist-test-slave-6wvm krb5kdc[24090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:16:13.948935 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:40657
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:40657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:14.059192 24103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:14.059453 24103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:14.059495 24103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:14.063537 24103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:14.063596 24103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:14.063616 24103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:14.063632 24103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:14.063647 24103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:14.068913 24103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:40657
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:40657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:14.069981 24103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:14.071069 24103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:16:14.079748 24103 server_base.cc:1034] running on GCE node
W20260924 16:16:14.079712 24112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:14.079710 24110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:14.079712 24109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:14.080572 24103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:14.081677 24103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:14.082921 24103 hybrid_clock.cc:648] HybridClock initialized: now 1790266574082889 us; error 48 us; skew 500 ppm
Sep 24 16:16:14 dist-test-slave-6wvm krb5kdc[24090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266574, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:14.086958 24103 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:16:14.088163 24103 webserver.cc:460] Webserver started at http://127.1.253.126:40677/ using document root <none> and password file <none>
I20260924 16:16:14.088922 24103 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:14.088989 24103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:14.089159 24103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:14.091235 24103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "d7f1f837286341f096b7c8b16dc76861"
format_stamp: "Formatted at 2026-09-24 16:16:14 on dist-test-slave-6wvm"
server_key: "4b82ad44857dc376b208928fb94f608e"
server_key_iv: "102427483c5ce43edfb3716582738f88"
server_key_version: "encryptionkey@0"
I20260924 16:16:14.091758 24103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "d7f1f837286341f096b7c8b16dc76861"
format_stamp: "Formatted at 2026-09-24 16:16:14 on dist-test-slave-6wvm"
server_key: "4b82ad44857dc376b208928fb94f608e"
server_key_iv: "102427483c5ce43edfb3716582738f88"
server_key_version: "encryptionkey@0"
I20260924 16:16:14.095428 24103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 16:16:14.097683 24119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:14.098977 24103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 16:16:14.099088 24103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "d7f1f837286341f096b7c8b16dc76861"
format_stamp: "Formatted at 2026-09-24 16:16:14 on dist-test-slave-6wvm"
server_key: "4b82ad44857dc376b208928fb94f608e"
server_key_iv: "102427483c5ce43edfb3716582738f88"
server_key_version: "encryptionkey@0"
I20260924 16:16:14.099221 24103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:14.103749 24103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:14.106873 24103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:14.107069 24103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:14.114033 24103 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:40657
I20260924 16:16:14.114060 24172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:40657 every 8 connection(s)
I20260924 16:16:14.114975 24103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:14.117771 24173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:14.123567 24173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861: Bootstrap starting.
I20260924 16:16:14.125216 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 24103
I20260924 16:16:14.125371 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:14.125701 2037 external_mini_cluster.cc:1428] Setting key 61a8876eaf57e95c9822b8a593654aa4
I20260924 16:16:14.126577 24173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:14.127564 24173 log.cc:824] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:14.129652 24173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861: No bootstrap required, opened a new log
I20260924 16:16:14.130879 24176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:14.126949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38505 (local address 127.1.253.126:40657)
0924 16:16:14.127413 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:14.127428 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:14.127464 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:16:14.128171 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:14.128190 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:14.128467 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:14.128814 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:14.129613 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:14.129618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:14.129630 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:14.129640 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:14.129710 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:14.129740 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:14.129868 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:14.129967 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 16:16:14.130137 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":146,"threads_started":1}
I20260924 16:16:14.133908 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:14.134141 24173 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f1f837286341f096b7c8b16dc76861" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 40657 } }
I20260924 16:16:14.134433 24173 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:14.134526 24173 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f1f837286341f096b7c8b16dc76861, State: Initialized, Role: FOLLOWER
I20260924 16:16:14.135044 24173 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f1f837286341f096b7c8b16dc76861" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 40657 } }
I20260924 16:16:14.135217 24173 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:14.135352 24173 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:14.135484 24173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:14.136859 24173 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f1f837286341f096b7c8b16dc76861" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 40657 } }
I20260924 16:16:14.137311 24173 leader_election.cc:302] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7f1f837286341f096b7c8b16dc76861; no voters:
I20260924 16:16:14.137720 24173 leader_election.cc:288] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:14.137817 24184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:14.137938 24176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:14.136158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55911 (local address 127.1.253.126:40657)
0924 16:16:14.136296 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:14.136302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:14.136449 (+ 147us) server_negotiation.cc:407] Connection header received
0924 16:16:14.136539 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:14.136542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:14.136586 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:14.136689 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:14.137427 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:14.137433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:14.137439 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:14.137455 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:14.137487 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:14.137503 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:14.137615 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:14.137686 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 16:16:14.137730 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:16:14.138150 24184 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [term 1 LEADER]: Becoming Leader. State: Replica: d7f1f837286341f096b7c8b16dc76861, State: Running, Role: LEADER
I20260924 16:16:14.138543 24184 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f1f837286341f096b7c8b16dc76861" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 40657 } }
I20260924 16:16:14.139317 24173 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:14.139926 24184 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7f1f837286341f096b7c8b16dc76861. Latest consensus state: current_term: 1 leader_uuid: "d7f1f837286341f096b7c8b16dc76861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f1f837286341f096b7c8b16dc76861" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 40657 } } }
I20260924 16:16:14.140100 24184 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:14.140396 24186 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7f1f837286341f096b7c8b16dc76861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f1f837286341f096b7c8b16dc76861" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 40657 } } }
I20260924 16:16:14.140518 24186 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:14.140563 24191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:14.143625 24191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:14.149238 24191 catalog_manager.cc:1347] Generated new cluster ID: 0f7f5cac3f914c148b2120d133af519d
I20260924 16:16:14.149314 24191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:14.158442 24191 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:14.158806 24191 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:14.173629 24191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7f1f837286341f096b7c8b16dc76861: Generated new TSK 0
I20260924 16:16:14.174538 24191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:14.180399 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 24103
2026-09-24T16:16:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:14 dist-test-slave-6wvm krb5kdc[24206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:14 dist-test-slave-6wvm krb5kdc[24206](info): set up 2 sockets
Sep 24 16:16:14 dist-test-slave-6wvm krb5kdc[24206](info): commencing operation
krb5kdc: starting...
W20260924 16:16:15.234397 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:15 dist-test-slave-6wvm krb5kdc[24206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:16:15.360984 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:37461
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:37461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:15.475064 24220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:15.475378 24220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:15.475454 24220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:15.479471 24220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:15.479564 24220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:15.479604 24220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:15.479655 24220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:15.479691 24220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:15.485001 24220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:37461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:37461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:15.486076 24220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:15.487178 24220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:15.496691 24226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:15.496691 24225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:15.496698 24228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:15.497174 24220 server_base.cc:1034] running on GCE node
I20260924 16:16:15.497772 24220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:15.499032 24220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:15.500200 24220 hybrid_clock.cc:648] HybridClock initialized: now 1790266575500188 us; error 29 us; skew 500 ppm
Sep 24 16:16:15 dist-test-slave-6wvm krb5kdc[24206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266575, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:15.503014 24220 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:16:15.504005 24220 webserver.cc:460] Webserver started at http://127.1.253.126:39659/ using document root <none> and password file <none>
I20260924 16:16:15.504606 24220 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:15.504710 24220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:15.504956 24220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:15.507341 24220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "4946c474dedd4afe8fe57aec682dc933"
format_stamp: "Formatted at 2026-09-24 16:16:15 on dist-test-slave-6wvm"
server_key: "dacf0ae61bb83e32265bc8df242b30b6"
server_key_iv: "7e82b639e33e5170143bf7656baf742d"
server_key_version: "encryptionkey@0"
I20260924 16:16:15.507990 24220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "4946c474dedd4afe8fe57aec682dc933"
format_stamp: "Formatted at 2026-09-24 16:16:15 on dist-test-slave-6wvm"
server_key: "dacf0ae61bb83e32265bc8df242b30b6"
server_key_iv: "7e82b639e33e5170143bf7656baf742d"
server_key_version: "encryptionkey@0"
I20260924 16:16:15.511984 24220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:16:15.514650 24235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:15.515970 24220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:15.516136 24220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "4946c474dedd4afe8fe57aec682dc933"
format_stamp: "Formatted at 2026-09-24 16:16:15 on dist-test-slave-6wvm"
server_key: "dacf0ae61bb83e32265bc8df242b30b6"
server_key_iv: "7e82b639e33e5170143bf7656baf742d"
server_key_version: "encryptionkey@0"
I20260924 16:16:15.516259 24220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:15.540158 24220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:15.543126 24220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:15.543330 24220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:15.550459 24220 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:37461
I20260924 16:16:15.550521 24288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:37461 every 8 connection(s)
I20260924 16:16:15.551488 24220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:15.554477 24289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:15.557708 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 24220
I20260924 16:16:15.557855 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:15.558143 2037 external_mini_cluster.cc:1428] Setting key f0e520cc319214180c71e2f50e011a9c
I20260924 16:16:15.560883 24289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933: Bootstrap starting.
I20260924 16:16:15.563488 24289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:15.564343 24289 log.cc:824] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:15.566587 24289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933: No bootstrap required, opened a new log
Sep 24 16:16:15 dist-test-slave-6wvm krb5kdc[24206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:16:15.571338 24289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4946c474dedd4afe8fe57aec682dc933" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 37461 } }
I20260924 16:16:15.571630 24289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:15.571682 24289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4946c474dedd4afe8fe57aec682dc933, State: Initialized, Role: FOLLOWER
I20260924 16:16:15.572182 24289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4946c474dedd4afe8fe57aec682dc933" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 37461 } }
I20260924 16:16:15.572381 24289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:15.572486 24289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:15.572606 24289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:15.573745 24289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4946c474dedd4afe8fe57aec682dc933" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 37461 } }
I20260924 16:16:15.573786 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:15.559721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53055 (local address 127.1.253.126:37461)
0924 16:16:15.560151 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:15.560167 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:15.560204 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:16:15.561250 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:15.561275 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:15.561724 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:15.562010 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:15.562964 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:15.564005 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:15.565102 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:15.565570 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:15.565697 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:15.568522 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:15.568529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:15.568545 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:15.568577 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:15.570633 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:15.571049 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:15.571056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:15.571061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:15.571200 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:15.571472 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:15.571475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:15.571477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:15.571993 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:15.572190 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:15.572461 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 16:16:15.572787 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":128,"threads_started":1}
I20260924 16:16:15.574143 24289 leader_election.cc:302] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4946c474dedd4afe8fe57aec682dc933; no voters:
I20260924 16:16:15.574514 24289 leader_election.cc:288] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:15.574719 24295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:15.575274 24295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [term 1 LEADER]: Becoming Leader. State: Replica: 4946c474dedd4afe8fe57aec682dc933, State: Running, Role: LEADER
I20260924 16:16:15.575765 24295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4946c474dedd4afe8fe57aec682dc933" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 37461 } }
I20260924 16:16:15.575855 24289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:15.577612 24298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4946c474dedd4afe8fe57aec682dc933. Latest consensus state: current_term: 1 leader_uuid: "4946c474dedd4afe8fe57aec682dc933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4946c474dedd4afe8fe57aec682dc933" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 37461 } } }
I20260924 16:16:15.577649 24296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4946c474dedd4afe8fe57aec682dc933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4946c474dedd4afe8fe57aec682dc933" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 37461 } } }
I20260924 16:16:15.577750 24298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:15.577756 24296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:15.578562 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 16:16:15.581562 24311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:16:15.581645 24311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:16:15.581710 24309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:15.582589 24309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:15.588317 24309 catalog_manager.cc:1347] Generated new cluster ID: 38d36a1a77d04bd7b160feb930335d80
I20260924 16:16:15.588379 24309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:15.592056 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:15.580738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50999 (local address 127.1.253.126:37461)
0924 16:16:15.580842 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:15.580846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:15.581097 (+ 251us) server_negotiation.cc:407] Connection header received
0924 16:16:15.581216 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:15.581219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:15.581267 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:15.581352 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:15.582187 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:15.583016 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:15.584129 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:15.584565 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:15.584677 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:15.586355 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:15.586359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:15.586361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:15.586387 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:15.588158 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:15.590750 (+ 2592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:15.590754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:15.590755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:15.590854 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:15.591178 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:15.591181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:15.591182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:15.591415 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:15.591527 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:15.591803 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 16:16:15.591911 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 16:16:15.605675 24309 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:15.606686 24309 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:15.613646 24309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4946c474dedd4afe8fe57aec682dc933: Generated new TSK 0
I20260924 16:16:15.614512 24309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:15.624325 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 24220
2026-09-24T16:16:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:15 dist-test-slave-6wvm krb5kdc[24322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:15 dist-test-slave-6wvm krb5kdc[24322](info): set up 2 sockets
Sep 24 16:16:15 dist-test-slave-6wvm krb5kdc[24322](info): commencing operation
krb5kdc: starting...
W20260924 16:16:16.666195 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:16 dist-test-slave-6wvm krb5kdc[24322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:16:16.793720 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:41549
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:41777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:41549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:16.906742 24335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:16.907032 24335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:16.907074 24335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:16.911118 24335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:16.911180 24335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:16.911201 24335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:16.911217 24335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:16.911232 24335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:16.916579 24335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:41777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:41549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:41549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:16.917599 24335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:16.918622 24335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:16.926910 24340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:16.926934 24341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:16.926990 24343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:16.927096 24335 server_base.cc:1034] running on GCE node
I20260924 16:16:16.927842 24335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:16.928918 24335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:16.930130 24335 hybrid_clock.cc:648] HybridClock initialized: now 1790266576930110 us; error 34 us; skew 500 ppm
Sep 24 16:16:16 dist-test-slave-6wvm krb5kdc[24322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266576, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:16.936820 24335 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:16:16.937785 24335 webserver.cc:460] Webserver started at http://127.1.253.126:46065/ using document root <none> and password file <none>
I20260924 16:16:16.938437 24335 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:16.938529 24335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:16.938767 24335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:16.940840 24335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "f751ce662af9493db1ee498ebc0e63d9"
format_stamp: "Formatted at 2026-09-24 16:16:16 on dist-test-slave-6wvm"
server_key: "070e4ef740bf867d240e71b9646eedb0"
server_key_iv: "3604849f1d917ab0a74b2384e3662f53"
server_key_version: "encryptionkey@0"
I20260924 16:16:16.941412 24335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "f751ce662af9493db1ee498ebc0e63d9"
format_stamp: "Formatted at 2026-09-24 16:16:16 on dist-test-slave-6wvm"
server_key: "070e4ef740bf867d240e71b9646eedb0"
server_key_iv: "3604849f1d917ab0a74b2384e3662f53"
server_key_version: "encryptionkey@0"
I20260924 16:16:16.945139 24335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 16:16:16.947655 24350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:16.948828 24335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:16:16.948974 24335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "f751ce662af9493db1ee498ebc0e63d9"
format_stamp: "Formatted at 2026-09-24 16:16:16 on dist-test-slave-6wvm"
server_key: "070e4ef740bf867d240e71b9646eedb0"
server_key_iv: "3604849f1d917ab0a74b2384e3662f53"
server_key_version: "encryptionkey@0"
I20260924 16:16:16.949083 24335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:16.968405 24335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:16.971976 24335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:16.972194 24335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:16.979813 24335 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:41549
I20260924 16:16:16.979851 24402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:41549 every 8 connection(s)
I20260924 16:16:16.980852 24335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:16.981069 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 24335
I20260924 16:16:16.981225 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:16.981622 2037 external_mini_cluster.cc:1428] Setting key 2d2464dd6a95ac570e245b934e44c79a
I20260924 16:16:16.984338 24403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:16.991257 24403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9: Bootstrap starting.
I20260924 16:16:16.991997 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:16.983178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52903 (local address 127.1.253.126:41549)
0924 16:16:16.983610 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:16.983630 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:16.983676 (+ 46us) server_negotiation.cc:407] Connection header received
0924 16:16:16.984543 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:16.984568 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:16.985015 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:16.985394 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:16.985806 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:16.987360 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:16.988298 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:16.988970 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:16.989140 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:16.989975 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:16.989982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:16.989999 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:16.990539 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:16.990609 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:16.990639 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:16.990741 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:16.990882 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 16:16:16.991088 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":113,"threads_started":1}
I20260924 16:16:16.993784 24403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:16.994637 24403 log.cc:824] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:16.994921 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:16.996647 24403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9: No bootstrap required, opened a new log
I20260924 16:16:17.001194 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:16.997067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40117 (local address 127.1.253.126:41549)
0924 16:16:16.997195 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:16.997199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:16.997351 (+ 152us) server_negotiation.cc:407] Connection header received
0924 16:16:16.997464 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:16.997466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:16.997504 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:16.997566 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:16.997868 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:16.998430 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:16.999619 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:16.999831 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:16.999906 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:17.000745 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:17.000747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:17.000749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:17.000760 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:17.000782 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:17.000792 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:17.000850 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:17.000980 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 16:16:17.001025 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:16:17.001371 24403 raft_consensus.cc:348] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f751ce662af9493db1ee498ebc0e63d9" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41549 } }
I20260924 16:16:17.001664 24403 raft_consensus.cc:374] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:17.001726 24403 raft_consensus.cc:729] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f751ce662af9493db1ee498ebc0e63d9, State: Initialized, Role: FOLLOWER
I20260924 16:16:17.002136 24403 consensus_queue.cc:260] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f751ce662af9493db1ee498ebc0e63d9" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41549 } }
I20260924 16:16:17.002321 24403 raft_consensus.cc:388] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:17.002393 24403 raft_consensus.cc:482] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:17.002499 24403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:17.003382 24403 raft_consensus.cc:504] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f751ce662af9493db1ee498ebc0e63d9" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41549 } }
I20260924 16:16:17.003739 24403 leader_election.cc:302] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f751ce662af9493db1ee498ebc0e63d9; no voters:
I20260924 16:16:17.004048 24403 leader_election.cc:288] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:17.004177 24414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:17.004420 24414 raft_consensus.cc:686] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [term 1 LEADER]: Becoming Leader. State: Replica: f751ce662af9493db1ee498ebc0e63d9, State: Running, Role: LEADER
I20260924 16:16:17.004781 24414 consensus_queue.cc:237] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f751ce662af9493db1ee498ebc0e63d9" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41549 } }
I20260924 16:16:17.005255 24403 sys_catalog.cc:564] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:17.006451 24416 sys_catalog.cc:455] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f751ce662af9493db1ee498ebc0e63d9. Latest consensus state: current_term: 1 leader_uuid: "f751ce662af9493db1ee498ebc0e63d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f751ce662af9493db1ee498ebc0e63d9" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41549 } } }
I20260924 16:16:17.006592 24416 sys_catalog.cc:458] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:17.006889 24415 sys_catalog.cc:455] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f751ce662af9493db1ee498ebc0e63d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f751ce662af9493db1ee498ebc0e63d9" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 41549 } } }
I20260924 16:16:17.007077 24415 sys_catalog.cc:458] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:17.006987 24423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:17.010056 24423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:17.015792 24423 catalog_manager.cc:1347] Generated new cluster ID: 5beba9da22e946dd92c60995e8a24c85
I20260924 16:16:17.015856 24423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:17.047042 24423 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:17.048038 24423 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:17.052496 24423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f751ce662af9493db1ee498ebc0e63d9: Generated new TSK 0
I20260924 16:16:17.053326 24423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:17.069051 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 24335
2026-09-24T16:16:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:17 dist-test-slave-6wvm krb5kdc[24435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:17 dist-test-slave-6wvm krb5kdc[24435](info): set up 2 sockets
Sep 24 16:16:17 dist-test-slave-6wvm krb5kdc[24435](info): commencing operation
krb5kdc: starting...
W20260924 16:16:18.134407 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:18 dist-test-slave-6wvm krb5kdc[24435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:16:18.262058 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:46775
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:46775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:18.377730 24448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:18.377992 24448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:18.378033 24448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:18.382026 24448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:18.382081 24448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:18.382099 24448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:18.382164 24448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:18.382194 24448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:18.387343 24448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:46775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:46775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:18.388368 24448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:18.389417 24448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:18.398000 24453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:18.397997 24454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:18.398031 24456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:18.398365 24448 server_base.cc:1034] running on GCE node
I20260924 16:16:18.398979 24448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:18.400089 24448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:18.401261 24448 hybrid_clock.cc:648] HybridClock initialized: now 1790266578401252 us; error 30 us; skew 500 ppm
Sep 24 16:16:18 dist-test-slave-6wvm krb5kdc[24435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266578, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:18.404070 24448 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:16:18.404989 24448 webserver.cc:460] Webserver started at http://127.1.253.126:40999/ using document root <none> and password file <none>
I20260924 16:16:18.405560 24448 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:18.405627 24448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:18.405920 24448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:18.408092 24448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "f08723da59f241cb8775c32f77ef6533"
format_stamp: "Formatted at 2026-09-24 16:16:18 on dist-test-slave-6wvm"
server_key: "9355007f1853f31ea72bd83b97bad0f3"
server_key_iv: "24dd4336c30326ed0067f228edbf793a"
server_key_version: "encryptionkey@0"
I20260924 16:16:18.408696 24448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "f08723da59f241cb8775c32f77ef6533"
format_stamp: "Formatted at 2026-09-24 16:16:18 on dist-test-slave-6wvm"
server_key: "9355007f1853f31ea72bd83b97bad0f3"
server_key_iv: "24dd4336c30326ed0067f228edbf793a"
server_key_version: "encryptionkey@0"
I20260924 16:16:18.412648 24448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:16:18.415202 24463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:18.416460 24448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:16:18.416589 24448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "f08723da59f241cb8775c32f77ef6533"
format_stamp: "Formatted at 2026-09-24 16:16:18 on dist-test-slave-6wvm"
server_key: "9355007f1853f31ea72bd83b97bad0f3"
server_key_iv: "24dd4336c30326ed0067f228edbf793a"
server_key_version: "encryptionkey@0"
I20260924 16:16:18.416697 24448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:18.421515 24448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:18.424849 24448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:18.424993 24448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:18.432785 24448 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:46775
I20260924 16:16:18.432790 24518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:46775 every 8 connection(s)
I20260924 16:16:18.433844 24448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:18.436837 24519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:18.438694 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 24448
I20260924 16:16:18.438848 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:18.439132 2037 external_mini_cluster.cc:1428] Setting key b97f2a553279d9348d01f211bd90fad9
I20260924 16:16:18.443974 24519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533: Bootstrap starting.
I20260924 16:16:18.444885 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:18.440614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52977 (local address 127.1.253.126:46775)
0924 16:16:18.441064 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:18.441084 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:18.441129 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:16:18.441975 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:18.442010 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:18.442421 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:18.442808 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:18.443485 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:18.443491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:18.443507 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:18.443518 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:18.443579 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:18.443606 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:18.443748 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:18.443819 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 16:16:18.443976 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":113,"threads_started":1}
I20260924 16:16:18.446547 24519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:18.447422 24519 log.cc:824] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:18.447840 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:18.449554 24519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533: No bootstrap required, opened a new log
I20260924 16:16:18.451637 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:18.449935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59535 (local address 127.1.253.126:46775)
0924 16:16:18.450070 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:18.450074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:18.450313 (+ 239us) server_negotiation.cc:407] Connection header received
0924 16:16:18.450401 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:18.450403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:18.450435 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:18.450499 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:18.451229 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:18.451232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:18.451235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:16:18.451245 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:18.451271 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:16:18.451281 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:18.451350 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:18.451448 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 16:16:18.451479 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:16:18.453965 24519 raft_consensus.cc:348] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08723da59f241cb8775c32f77ef6533" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46775 } }
I20260924 16:16:18.454223 24519 raft_consensus.cc:374] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:18.454310 24519 raft_consensus.cc:729] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08723da59f241cb8775c32f77ef6533, State: Initialized, Role: FOLLOWER
I20260924 16:16:18.454736 24519 consensus_queue.cc:260] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08723da59f241cb8775c32f77ef6533" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46775 } }
I20260924 16:16:18.454929 24519 raft_consensus.cc:388] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:18.455026 24519 raft_consensus.cc:482] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:18.455130 24519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:18.456158 24519 raft_consensus.cc:504] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08723da59f241cb8775c32f77ef6533" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46775 } }
I20260924 16:16:18.456565 24519 leader_election.cc:302] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f08723da59f241cb8775c32f77ef6533; no voters:
I20260924 16:16:18.456920 24519 leader_election.cc:288] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:18.457042 24529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:18.457290 24529 raft_consensus.cc:686] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [term 1 LEADER]: Becoming Leader. State: Replica: f08723da59f241cb8775c32f77ef6533, State: Running, Role: LEADER
I20260924 16:16:18.457660 24529 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08723da59f241cb8775c32f77ef6533" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46775 } }
I20260924 16:16:18.458297 24519 sys_catalog.cc:564] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:18.459589 24530 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f08723da59f241cb8775c32f77ef6533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08723da59f241cb8775c32f77ef6533" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46775 } } }
I20260924 16:16:18.459571 24531 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f08723da59f241cb8775c32f77ef6533. Latest consensus state: current_term: 1 leader_uuid: "f08723da59f241cb8775c32f77ef6533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08723da59f241cb8775c32f77ef6533" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 46775 } } }
I20260924 16:16:18.459728 24530 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:18.459728 24531 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533 [sys.catalog]: This master's current role is: LEADER
W20260924 16:16:18.463028 24545 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:16:18.463095 24545 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:16:18.463183 24540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:18.463861 24540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:18.468842 24540 catalog_manager.cc:1347] Generated new cluster ID: e5dcd7adb2da4064a18c4fb59f628825
I20260924 16:16:18.468901 24540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:18.487952 24540 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:18.488327 24540 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:18.507387 24540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f08723da59f241cb8775c32f77ef6533: Generated new TSK 0
I20260924 16:16:18.508167 24540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:16:18.518465 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 24448
2026-09-24T16:16:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1449 ms)
[----------] 18 tests from AuthTokenIssuingTest (25933 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:18 dist-test-slave-6wvm krb5kdc[24551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:18 dist-test-slave-6wvm krb5kdc[24551](info): set up 2 sockets
Sep 24 16:16:18 dist-test-slave-6wvm krb5kdc[24551](info): commencing operation
krb5kdc: starting...
W20260924 16:16:19.619196 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:19 dist-test-slave-6wvm krb5kdc[24551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:16:19.743382 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:43889
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:43889,127.1.253.125:33429,127.1.253.124:45283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:19.854218 24565 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:16:19.854596 24565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:19.854698 24565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:19.854794 24565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:19.858780 24565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:19.858886 24565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:19.858923 24565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:19.858969 24565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:19.859004 24565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:19.864311 24565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:43889,127.1.253.125:33429,127.1.253.124:45283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:43889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:19.865513 24565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:19.866650 24565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:19.875597 24573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:19.875590 24570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:19.875586 24571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:19.875846 24565 server_base.cc:1034] running on GCE node
I20260924 16:16:19.876498 24565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:19.877636 24565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:19.878899 24565 hybrid_clock.cc:648] HybridClock initialized: now 1790266579878862 us; error 45 us; skew 500 ppm
Sep 24 16:16:19 dist-test-slave-6wvm krb5kdc[24551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266579, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:19.881670 24565 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:16:19.882627 24565 webserver.cc:460] Webserver started at http://127.1.253.126:38127/ using document root <none> and password file <none>
I20260924 16:16:19.883205 24565 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:19.883297 24565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:19.883530 24565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:19.885598 24565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "b3212c0090954817961367ed0678e0f4"
format_stamp: "Formatted at 2026-09-24 16:16:19 on dist-test-slave-6wvm"
server_key: "71a04a572a2d5997781d4d85ace384cb"
server_key_iv: "ac9d6c5cff89b4dd449ec52d5832dcdb"
server_key_version: "encryptionkey@0"
I20260924 16:16:19.886215 24565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "b3212c0090954817961367ed0678e0f4"
format_stamp: "Formatted at 2026-09-24 16:16:19 on dist-test-slave-6wvm"
server_key: "71a04a572a2d5997781d4d85ace384cb"
server_key_iv: "ac9d6c5cff89b4dd449ec52d5832dcdb"
server_key_version: "encryptionkey@0"
I20260924 16:16:19.889945 24565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:19.892581 24580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:19.893826 24565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:16:19.893970 24565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "b3212c0090954817961367ed0678e0f4"
format_stamp: "Formatted at 2026-09-24 16:16:19 on dist-test-slave-6wvm"
server_key: "71a04a572a2d5997781d4d85ace384cb"
server_key_iv: "ac9d6c5cff89b4dd449ec52d5832dcdb"
server_key_version: "encryptionkey@0"
I20260924 16:16:19.894083 24565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:19.905074 24565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:19.907927 24565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:19.908156 24565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:19.915228 24565 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:43889
I20260924 16:16:19.915550 24632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:43889 every 8 connection(s)
I20260924 16:16:19.916265 24565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:19.919268 24633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:19.920308 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 24565
I20260924 16:16:19.920442 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:19.920734 2037 external_mini_cluster.cc:1428] Setting key 5b8a607d000773bd523767af86c9aee1
I20260924 16:16:19.923767 24633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.253.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.125@KRBTEST.COM" created.
Sep 24 16:16:19 dist-test-slave-6wvm krb5kdc[24551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266579, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:16:19.940301 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:19.926150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:43889 (local address 127.0.0.1:58270)
0924 16:16:19.926780 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:19.926843 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:19.928085 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:19.928886 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:19.928893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:19.929395 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:19.929781 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:19.929803 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:19.930996 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:19.930999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:19.931822 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:19.931832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:19.932540 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:19.932544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:19.932660 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:19.933260 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:19.933277 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:19.934978 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:19.937387 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:19.937396 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:19.937404 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:19.937623 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:19.938048 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:19.938050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:19.938052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:19.938154 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:19.938999 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:19.939005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:19.939129 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 16:16:19.939270 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 16:16:19.939519 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":141,"threads_started":1}
I20260924 16:16:19.940303 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:19.926150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.1.253.126:43889)
0924 16:16:19.926727 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:19.926745 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:19.926974 (+ 229us) server_negotiation.cc:407] Connection header received
0924 16:16:19.928432 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:19.928436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:19.928638 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:19.928811 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:19.929944 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:19.930881 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:19.931929 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:19.932354 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:19.932554 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:19.935152 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:19.935161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:19.935177 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:19.935209 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:19.937255 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:19.937743 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:19.937751 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:19.937753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:19.937926 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:19.938275 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:19.938281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:19.938282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:19.938840 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:19.938979 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:19.939395 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 16:16:19.939694 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":455,"spinlock_wait_cycles":58240,"thread_start_us":151,"threads_started":1}
I20260924 16:16:19.942816 24633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:16:19.943552 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:19.943315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:33429 (local address 127.0.0.1:37566)
0924 16:16:19.943427 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:19.943493 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 16:16:19.943738 24582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.253.125:33429: connect: Connection refused (error 111)
I20260924 16:16:19.944515 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:19.944296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:33429 (local address 127.0.0.1:37570)
0924 16:16:19.944405 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:19.944457 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:16:19.944651 24633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.125:33429: Network error: Client connection negotiation failed: client connection to 127.1.253.125:33429: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.125.keytab.
Entry for principal kudu/127.1.253.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.125.keytab.
I20260924 16:16:19.949572 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.125:33429
--webserver_interface=127.1.253.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.125.keytab
--principal=kudu/127.1.253.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:43889,127.1.253.125:33429,127.1.253.124:45283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:16:19.994007 24633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } attempt: 1
I20260924 16:16:19.995024 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:19.994655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:33429 (local address 127.0.0.1:37584)
0924 16:16:19.994840 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:19.994920 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 16:16:19.995797 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:19.995563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:33429 (local address 127.0.0.1:37588)
0924 16:16:19.995675 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:19.995729 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 16:16:19.995934 24633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.125:33429: Network error: Client connection negotiation failed: client connection to 127.1.253.125:33429: connect: Connection refused (error 111)
W20260924 16:16:20.063673 24640 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:16:20.063939 24640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:20.063962 24640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:20.063998 24640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 16:16:20.064153 24633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } attempt: 2
I20260924 16:16:20.065176 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.064717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:33429 (local address 127.0.0.1:37590)
0924 16:16:20.064960 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.065048 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 16:16:20.065892 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.065685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:33429 (local address 127.0.0.1:37594)
0924 16:16:20.065788 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.065846 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:33429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 16:16:20.066042 24633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.125:33429: Network error: Client connection negotiation failed: client connection to 127.1.253.125:33429: connect: Connection refused (error 111)
W20260924 16:16:20.068704 24640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:20.068782 24640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:20.068828 24640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:20.068882 24640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:20.068918 24640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:20.074141 24640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:43889,127.1.253.125:33429,127.1.253.124:45283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.125.keytab
--principal=kudu/127.1.253.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.125:33429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.253.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:20.075192 24640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:20.076229 24640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:20.084625 24646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:20.084627 24645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:20.084753 24648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:20.086462 24640 server_base.cc:1034] running on GCE node
I20260924 16:16:20.087008 24640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:20.088553 24640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:20.089756 24640 hybrid_clock.cc:648] HybridClock initialized: now 1790266580089727 us; error 39 us; skew 500 ppm
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266580, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:20.093430 24640 init.cc:377] Logged in from keytab as kudu/127.1.253.125@KRBTEST.COM (short username kudu)
I20260924 16:16:20.094456 24640 webserver.cc:460] Webserver started at http://127.1.253.125:44533/ using document root <none> and password file <none>
I20260924 16:16:20.095050 24640 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:20.095140 24640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:20.095355 24640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:20.097548 24640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/data/instance:
uuid: "a490df74eaea4ab08581c2a816242e6c"
format_stamp: "Formatted at 2026-09-24 16:16:20 on dist-test-slave-6wvm"
server_key: "07999c7a635fc5f7e118cebc5ddea5b1"
server_key_iv: "8c7780eb95d3148e833d86671e1a9847"
server_key_version: "encryptionkey@0"
I20260924 16:16:20.098169 24640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/wal/instance:
uuid: "a490df74eaea4ab08581c2a816242e6c"
format_stamp: "Formatted at 2026-09-24 16:16:20 on dist-test-slave-6wvm"
server_key: "07999c7a635fc5f7e118cebc5ddea5b1"
server_key_iv: "8c7780eb95d3148e833d86671e1a9847"
server_key_version: "encryptionkey@0"
I20260924 16:16:20.102313 24640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:16:20.104816 24655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:20.106103 24640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 16:16:20.106292 24640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/wal
uuid: "a490df74eaea4ab08581c2a816242e6c"
format_stamp: "Formatted at 2026-09-24 16:16:20 on dist-test-slave-6wvm"
server_key: "07999c7a635fc5f7e118cebc5ddea5b1"
server_key_iv: "8c7780eb95d3148e833d86671e1a9847"
server_key_version: "encryptionkey@0"
I20260924 16:16:20.106401 24640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:20.121205 24640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:20.124056 24640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:20.124262 24640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:20.131667 24640 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.125:33429
I20260924 16:16:20.131973 24707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.125:33429 every 8 connection(s)
I20260924 16:16:20.132696 24640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
I20260924 16:16:20.135522 24708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:20.136300 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 24640
I20260924 16:16:20.136428 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-1/wal/instance
I20260924 16:16:20.136741 2037 external_mini_cluster.cc:1428] Setting key 2db3b6504975efddcb32e49677f48f9b
I20260924 16:16:20.139384 24708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.253.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.124@KRBTEST.COM" created.
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266580, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.125@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:16:20.154088 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.141298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.1.253.126:43889)
0924 16:16:20.141463 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.141469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.142358 (+ 889us) server_negotiation.cc:407] Connection header received
0924 16:16:20.143315 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.143320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.143391 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.143501 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.144636 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.145302 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.146512 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.146760 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.146873 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.149543 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.149546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.149549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.149585 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.151752 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.152416 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.152418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.152420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.152493 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.152921 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.152923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.152925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.153099 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.153179 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.153850 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.153955 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:16:20.155387 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.141680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:43889 (local address 127.0.0.1:58280)
0924 16:16:20.142163 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.142216 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:20.143083 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:20.143658 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:20.143664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:20.143981 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:20.144471 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.144484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.145422 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.145425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.146360 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.146371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.146914 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.146917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.147088 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.147717 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:20.147733 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:20.149361 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:20.151880 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.151888 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.151905 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.152310 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.152591 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.152597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.152602 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.152811 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.153215 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:20.153223 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:20.153571 (+ 348us) client_negotiation.cc:769] Sending connection context
0924 16:16:20.153800 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 16:16:20.154184 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":133,"threads_started":1}
I20260924 16:16:20.156850 24708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:16:20.157193 24633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } attempt: 3
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266579, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.125@KRBTEST.COM
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266580, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.125@KRBTEST.COM for kudu/127.1.253.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.124.keytab.
Entry for principal kudu/127.1.253.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.124.keytab.
I20260924 16:16:20.166294 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.124:45283
--webserver_interface=127.1.253.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.124.keytab
--principal=kudu/127.1.253.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:42821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:43889,127.1.253.125:33429,127.1.253.124:45283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:16:20.169333 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.157600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:33429 (local address 127.0.0.1:37604)
0924 16:16:20.157706 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.157720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:20.157793 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:20.158519 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:20.158523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:20.158684 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:20.158802 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.158808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.160325 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.160328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.161199 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.161210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.162202 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.162206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.162382 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.163024 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:20.163042 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:20.164367 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:20.167564 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.167567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.167569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.167806 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.168146 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.168150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.168152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.168245 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.168738 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:20.168741 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:20.168844 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 16:16:20.168983 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 16:16:20.169142 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 16:16:20.169337 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.157467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:33429 (local address 127.0.0.1:37598)
0924 16:16:20.157602 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.157616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:20.157696 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:20.158465 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:20.158472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:20.158651 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:20.158790 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.158797 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.160325 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.160328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.161199 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.161210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.162202 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.162206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.162340 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.162921 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:20.162933 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:20.164816 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:20.167547 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.167550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.167552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.167801 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.168207 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.168212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.168217 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.168304 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.168822 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:20.168826 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:20.168916 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 16:16:20.169020 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:16:20.169185 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 16:16:20.169476 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.157994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.1.253.125:33429)
0924 16:16:20.158303 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.158312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.158324 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:16:20.158357 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.158362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.158405 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.158479 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.158929 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.160205 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.161318 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.162045 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.162186 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.164604 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.164620 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.164634 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.164657 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.167449 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.167929 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.167937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.167939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.168045 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.168340 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.168344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.168346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.168624 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.168776 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.169094 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.169253 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":409,"server-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260924 16:16:20.169862 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.157645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.1.253.125:33429)
0924 16:16:20.157997 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.158005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.158024 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:16:20.158058 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.158062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.158302 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.158446 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.158929 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.160205 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.161318 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.162045 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.162191 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.165003 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.165009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.165014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.165039 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.167426 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.167932 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.167942 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.167944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.168083 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.168398 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.168402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.168403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.168651 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.168778 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.169589 (+ 811us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.169712 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":185,"server-negotiator.queue_time_us":254,"spinlock_wait_cycles":9088,"thread_start_us":151,"threads_started":1}
I20260924 16:16:20.171383 24633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:16:20.171567 24708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:16:20.172242 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.171990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57786)
0924 16:16:20.172127 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.172182 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 16:16:20.172379 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.172045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57796)
0924 16:16:20.172221 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.172302 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260924 16:16:20.172595 24658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
I20260924 16:16:20.173038 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.172764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57810)
0924 16:16:20.172919 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.172969 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 16:16:20.173197 24633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:45283: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
I20260924 16:16:20.173832 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.173638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57822)
0924 16:16:20.173745 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.173786 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 16:16:20.174001 24708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:45283: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
I20260924 16:16:20.204358 24633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } attempt: 1
I20260924 16:16:20.205236 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.204915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57836)
0924 16:16:20.205074 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.205155 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 16:16:20.205981 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.205751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57844)
0924 16:16:20.205863 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.205911 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 16:16:20.206106 24633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:45283: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
I20260924 16:16:20.223237 24708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } attempt: 1
I20260924 16:16:20.224153 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.223854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57854)
0924 16:16:20.224011 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.224075 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 16:16:20.224972 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.224711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57870)
0924 16:16:20.224826 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.224886 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 16:16:20.225106 24708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:45283: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
I20260924 16:16:20.281277 24633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } attempt: 2
I20260924 16:16:20.282203 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.281888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57876)
0924 16:16:20.282052 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.282124 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 16:16:20.282979 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.282765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57884)
0924 16:16:20.282874 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.282926 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 16:16:20.283107 24633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:45283: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
W20260924 16:16:20.286571 24715 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:16:20.286899 24715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:20.286962 24715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:20.287048 24715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:20.291129 24715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:20.291204 24715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:20.291250 24715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:20.291308 24715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:20.291345 24715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:20.293251 24708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } attempt: 2
I20260924 16:16:20.294077 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.293781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57892)
0924 16:16:20.293934 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.294003 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 16:16:20.294898 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.294676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57898)
0924 16:16:20.294790 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.294841 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 16:16:20.295077 24708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:45283: Network error: Client connection negotiation failed: client connection to 127.1.253.124:45283: connect: Connection refused (error 111)
I20260924 16:16:20.297196 24715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:42821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:43889,127.1.253.125:33429,127.1.253.124:45283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.124.keytab
--principal=kudu/127.1.253.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.124:45283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.253.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:20.298494 24715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:20.299636 24715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:20.308470 24721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:20.308476 24722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:20.308513 24715 server_base.cc:1034] running on GCE node
W20260924 16:16:20.308476 24724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:20.309399 24715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:20.310544 24715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:20.311705 24715 hybrid_clock.cc:648] HybridClock initialized: now 1790266580311691 us; error 24 us; skew 500 ppm
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266580, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:20.314462 24715 init.cc:377] Logged in from keytab as kudu/127.1.253.124@KRBTEST.COM (short username kudu)
I20260924 16:16:20.315410 24715 webserver.cc:460] Webserver started at http://127.1.253.124:34253/ using document root <none> and password file <none>
I20260924 16:16:20.315995 24715 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:20.316061 24715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:20.316293 24715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:20.318444 24715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/data/instance:
uuid: "591fd4e05e9c4307b8c5091ab652b584"
format_stamp: "Formatted at 2026-09-24 16:16:20 on dist-test-slave-6wvm"
server_key: "c35ee4f2107cede2dd3ecd81851891b2"
server_key_iv: "6724882806f9b7ee067c925f2c8a9c25"
server_key_version: "encryptionkey@0"
I20260924 16:16:20.319025 24715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/wal/instance:
uuid: "591fd4e05e9c4307b8c5091ab652b584"
format_stamp: "Formatted at 2026-09-24 16:16:20 on dist-test-slave-6wvm"
server_key: "c35ee4f2107cede2dd3ecd81851891b2"
server_key_iv: "6724882806f9b7ee067c925f2c8a9c25"
server_key_version: "encryptionkey@0"
I20260924 16:16:20.322837 24715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:16:20.325286 24731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:20.326610 24715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:16:20.326746 24715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/wal
uuid: "591fd4e05e9c4307b8c5091ab652b584"
format_stamp: "Formatted at 2026-09-24 16:16:20 on dist-test-slave-6wvm"
server_key: "c35ee4f2107cede2dd3ecd81851891b2"
server_key_iv: "6724882806f9b7ee067c925f2c8a9c25"
server_key_version: "encryptionkey@0"
I20260924 16:16:20.326874 24715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:20.352027 24715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:20.356061 24715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:20.356231 24715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:20.363318 24715 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.124:45283
I20260924 16:16:20.363628 24784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.124:45283 every 8 connection(s)
I20260924 16:16:20.364356 24715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
I20260924 16:16:20.366328 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 24715
I20260924 16:16:20.366460 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266297063660-2037-0/minicluster-data/master-2/wal/instance
I20260924 16:16:20.366748 2037 external_mini_cluster.cc:1428] Setting key e974ced83a56c7c8f714e7abaf32bb98
I20260924 16:16:20.367630 24785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:20.372661 24785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.124@KRBTEST.COM
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266580, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.124@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:16:20.382301 24633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } attempt: 3
I20260924 16:16:20.382694 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.368214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.1.253.124:45283)
0924 16:16:20.368655 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.368681 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.368725 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:16:20.369657 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.369681 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.370110 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.370499 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.371237 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.372782 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.373676 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.374354 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.374492 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.377231 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.377237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.377246 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.377272 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.379221 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.379736 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.379741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.379746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.379855 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.380229 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.380232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.380233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.380621 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.380780 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.381126 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.381434 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260924 16:16:20.384872 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.374755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.1.253.126:43889)
0924 16:16:20.375302 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.375308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.375324 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:16:20.375396 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.375399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.375451 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.375540 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.376280 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.376824 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.377885 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.378120 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.378232 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.381288 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.381292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.381295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.381323 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.382875 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.383463 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.383466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.383468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.383563 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.383873 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.383879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.383881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.384136 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.384249 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.384602 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.384709 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466}
I20260924 16:16:20.384891 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.374688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:43889 (local address 127.0.0.1:58296)
0924 16:16:20.375133 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.375192 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:20.375303 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:20.375613 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:20.375621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:20.376024 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:20.376151 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.376157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.376942 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.376947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.377745 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.377761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.378258 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.378263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.378389 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.378884 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:20.378896 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:20.381126 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:20.383002 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.383010 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.383016 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.383344 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.383670 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.383673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.383675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.383775 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.384249 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:20.384256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:20.384372 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 16:16:20.384527 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 16:16:20.384709 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":156,"threads_started":1}
I20260924 16:16:20.386163 24785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:16:20.386226 24708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } attempt: 3
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266579, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.124@KRBTEST.COM
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:16:20.392280 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.382823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.1.253.124:45283)
0924 16:16:20.383039 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.383043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.383056 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:16:20.383182 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.383185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.383233 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.383787 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.383876 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.384582 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.385741 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.386073 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.386513 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.388971 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.388974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.388977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.388999 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.390593 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.391050 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.391053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.391054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.391148 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.391454 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.391457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.391460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.391661 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.391787 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.392039 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.392150 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":106,"threads_started":1}
I20260924 16:16:20.392277 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.382778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57920)
0924 16:16:20.382978 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.383006 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:20.383087 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:20.383374 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:20.383380 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:20.383556 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:20.383678 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.383686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.384731 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.384736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.385443 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.385454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.386224 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.386228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.386340 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.387359 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:20.387380 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:20.388861 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:20.390701 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.390704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.390706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.390932 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.391254 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.391257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.391260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.391345 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.391782 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:20.391786 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:20.391882 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 16:16:20.391989 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:16:20.392113 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266580, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.125@KRBTEST.COM for kudu/127.1.253.124@KRBTEST.COM
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266580, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.124@KRBTEST.COM for kudu/127.1.253.125@KRBTEST.COM
I20260924 16:16:20.396306 24633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4: Bootstrap starting.
I20260924 16:16:20.396564 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.385607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.1.253.126:43889)
0924 16:16:20.385741 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.385745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.385756 (+ 11us) server_negotiation.cc:407] Connection header received
0924 16:16:20.385784 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.385787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.385821 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.385888 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.386742 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.387360 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.388394 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.388695 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.388794 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.392794 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.392797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.392800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.392824 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.394537 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.395034 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.395038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.395042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.395132 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.395440 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.395443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.395445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.395641 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.395803 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.396195 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.396313 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54,"spinlock_wait_cycles":896}
I20260924 16:16:20.398911 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.386792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:33429 (local address 127.0.0.1:37616)
0924 16:16:20.387964 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.387981 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:20.388074 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:20.388308 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:20.388311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:20.388468 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:20.388595 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.388606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.389597 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.389600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.390543 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.390556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.391297 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.391303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.391410 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.392932 (+ 1522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:20.392947 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:20.394690 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:20.396810 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.396814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.396816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.397062 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.397383 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.397386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.397388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.397479 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.398325 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:20.398328 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:20.398500 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 16:16:20.398617 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 16:16:20.398755 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096}
I20260924 16:16:20.398936 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.387031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37616 (local address 127.1.253.125:33429)
0924 16:16:20.387175 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.387180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.388064 (+ 884us) server_negotiation.cc:407] Connection header received
0924 16:16:20.388144 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.388147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.388191 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.388266 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.388698 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.389460 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.390701 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.391077 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.391150 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.394853 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.394860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.394864 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.394894 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.396575 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.397172 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.397175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.397177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.397269 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.397584 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.397588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.397589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.397886 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.398004 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.398658 (+ 654us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.398764 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56,"spinlock_wait_cycles":57600}
I20260924 16:16:20.399636 24633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:20.400462 24633 log.cc:824] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:20.400964 24785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:16:20.403136 24633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4: No bootstrap required, opened a new log
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.125@KRBTEST.COM
Sep 24 16:16:20 dist-test-slave-6wvm krb5kdc[24551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266580, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.124@KRBTEST.COM for kudu/127.1.253.124@KRBTEST.COM
I20260924 16:16:20.407989 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.386693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57922)
0924 16:16:20.386972 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.386988 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:20.387070 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:20.387471 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:20.387474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:20.387654 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:20.387758 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.387765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.390626 (+ 2861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.390633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.391460 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.391472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.392159 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.392166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.392266 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.392683 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:20.392698 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:20.394948 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:20.405470 (+ 10522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.405474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.405476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.406036 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.406557 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.406560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.406562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.406681 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.407452 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:20.407455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:20.407537 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 16:16:20.407645 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:16:20.407804 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260924 16:16:20.408138 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.397986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.1.253.125:33429)
0924 16:16:20.398106 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.398117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.398168 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:16:20.398325 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.398328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.398372 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.398455 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.399175 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.399896 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.400681 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.400925 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.401038 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.404024 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.404028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.404031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.404056 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.405757 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.406546 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.406549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.406551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.406788 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.407091 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.407094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.407096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.407337 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.407547 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.407888 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.407994 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 16:16:20.407989 24633 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } }
I20260924 16:16:20.408278 24633 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3212c0090954817961367ed0678e0f4, State: Initialized, Role: FOLLOWER
I20260924 16:16:20.408665 24633 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } }
I20260924 16:16:20.408959 24790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.386953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.1.253.124:45283)
0924 16:16:20.387228 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.387239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.387252 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:16:20.387286 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.387289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.387340 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.387431 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.388980 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.390299 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.391632 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.392022 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.402293 (+ 10271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.402631 (+ 338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.402635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.402638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.402653 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.405102 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.406350 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.406354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.406356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.406449 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.407105 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.407108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.407110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.407338 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.408334 (+ 996us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.408416 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.408534 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":226,"server-negotiator.queue_time_us":193}
I20260924 16:16:20.410389 24633 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:20.411248 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.401504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:45283 (local address 127.0.0.1:57930)
0924 16:16:20.401643 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:20.401658 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:20.401730 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:20.402018 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:20.402021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:20.402178 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:20.402416 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.402423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.403404 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.403409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.404206 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:20.404218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.404932 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.404938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:20.405016 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.405414 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:20.405433 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:20.406933 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:20.409672 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.409675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.409676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.409904 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.410177 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:20.410180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:20.410181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:20.410285 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:20.410758 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:20.410762 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:20.410848 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 16:16:20.410933 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 16:16:20.411074 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":132}
I20260924 16:16:20.411870 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.401682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57930 (local address 127.1.253.124:45283)
0924 16:16:20.401792 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.401799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.401815 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:16:20.401851 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.401854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.401899 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.401986 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.402536 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.403284 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.404330 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.404696 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.404807 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.407084 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.407090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.407093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.407118 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.409559 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.410014 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.410017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.410018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.410091 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.410376 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.410379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.410380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.410634 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.410723 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.411618 (+ 895us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.411724 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":292,"server-negotiator.queue_time_us":40}
I20260924 16:16:20.412253 24708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c: Bootstrap starting.
I20260924 16:16:20.414791 24785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584: Bootstrap starting.
W20260924 16:16:20.416404 24803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:16:20.416482 24803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:16:20.417410 24785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:20.418354 24708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:20.418360 24785 log.cc:824] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:20.419143 24708 log.cc:824] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:20.419730 24792 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } } }
I20260924 16:16:20.419862 24792 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:16:20.421439 24708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c: No bootstrap required, opened a new log
I20260924 16:16:20.417533 24565 master_runner.cc:186] Error getting master registration for 127.1.253.125:33429: OK, instance_id { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" instance_seqno: 1790266580129632 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:16:20.421643 24565 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:16:20.421645 24785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584: No bootstrap required, opened a new log
I20260924 16:16:20.421747 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:20.423231 24607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b3212c0090954817961367ed0678e0f4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58308
I20260924 16:16:20.423447 24607 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 16:16:20.423571 24607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:20.424983 24607 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } }
I20260924 16:16:20.425031 24785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } }
I20260924 16:16:20.425203 24785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591fd4e05e9c4307b8c5091ab652b584, State: Initialized, Role: FOLLOWER
I20260924 16:16:20.425577 24785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } }
I20260924 16:16:20.425940 24607 leader_election.cc:288] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [CANDIDATE]: Term 1 election: Requested vote from peers a490df74eaea4ab08581c2a816242e6c (127.1.253.125:33429), 591fd4e05e9c4307b8c5091ab652b584 (127.1.253.124:45283)
I20260924 16:16:20.427070 24785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:20.427160 24759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3212c0090954817961367ed0678e0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "591fd4e05e9c4307b8c5091ab652b584"
I20260924 16:16:20.427363 24759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:20.429334 24759 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3212c0090954817961367ed0678e0f4 in term 1.
I20260924 16:16:20.429893 24583 leader_election.cc:302] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 591fd4e05e9c4307b8c5091ab652b584, b3212c0090954817961367ed0678e0f4; no voters:
I20260924 16:16:20.430414 24792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:20.430656 24792 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [term 1 LEADER]: Becoming Leader. State: Replica: b3212c0090954817961367ed0678e0f4, State: Running, Role: LEADER
I20260924 16:16:20.431006 24792 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } }
I20260924 16:16:20.431783 24708 raft_consensus.cc:348] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } }
I20260924 16:16:20.432116 24708 raft_consensus.cc:729] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a490df74eaea4ab08581c2a816242e6c, State: Initialized, Role: FOLLOWER
I20260924 16:16:20.432476 24708 consensus_queue.cc:260] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } }
I20260924 16:16:20.432845 24792 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3212c0090954817961367ed0678e0f4. Latest consensus state: current_term: 1 leader_uuid: "b3212c0090954817961367ed0678e0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } } }
I20260924 16:16:20.432964 24792 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:20.433899 24708 sys_catalog.cc:564] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:20.433964 24682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3212c0090954817961367ed0678e0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a490df74eaea4ab08581c2a816242e6c"
I20260924 16:16:20.434221 24682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:20.429847 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } } }
I20260924 16:16:20.434551 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 16:16:20.433058 24823 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:16:20.435438 24823 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:16:20.435659 24682 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3212c0090954817961367ed0678e0f4 in term 1.
I20260924 16:16:20.440104 24828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:20.437798 24715 master_runner.cc:186] Error getting master registration for 127.1.253.125:33429: OK, instance_id { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" instance_seqno: 1790266580129632 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:16:20.441143 24715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:16:20.441291 24828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260924 16:16:20.446422 24842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 16:16:20.446776 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.434349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.1.253.125:33429)
0924 16:16:20.434483 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.434488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.437323 (+ 2835us) server_negotiation.cc:407] Connection header received
0924 16:16:20.437543 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.437548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.437603 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.437674 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.438794 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.439481 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.440600 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.440947 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.441021 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.443034 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.443037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.443040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.443067 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.444866 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.445361 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.445365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.445366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.445450 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.445758 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.445761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.445763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.445974 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.446083 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.446445 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.446568 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:16:20.446672 24827 sys_catalog.cc:455] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } } }
I20260924 16:16:20.448438 24827 sys_catalog.cc:458] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [sys.catalog]: This master's current role is: FOLLOWER
W20260924 16:16:20.448557 24842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:16:20.452991 24790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.436536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.1.253.124:45283)
0924 16:16:20.436650 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.436655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.442666 (+ 6011us) server_negotiation.cc:407] Connection header received
0924 16:16:20.442917 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.442920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.442968 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.443018 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.443818 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.444948 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.446036 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.446429 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.446499 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.449042 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.449045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.449048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.449074 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.450898 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.451406 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.451411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.451414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.451516 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.451868 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.451873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.451876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.452159 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.452300 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.452656 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.452787 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:16:20.456121 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:20.436962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.1.253.126:43889)
0924 16:16:20.437072 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.437076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.447100 (+ 10024us) server_negotiation.cc:407] Connection header received
0924 16:16:20.447408 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.447415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.447474 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.447531 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.448369 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.449068 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.450488 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:20.450862 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:20.450965 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:20.452572 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:20.452577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:20.452583 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:20.452628 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:20.454341 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.454838 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.454842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.454843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.454930 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:20.455208 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:20.455211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:20.455213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:20.455389 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:20.455539 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:20.455797 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 16:16:20.455910 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:16:20.461522 24759 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [term 1 FOLLOWER]: Refusing update from remote peer b3212c0090954817961367ed0678e0f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:16:20.461741 24682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [term 1 FOLLOWER]: Refusing update from remote peer b3212c0090954817961367ed0678e0f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:16:20.462388 24792 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:16:20.462602 24824 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:16:20.467074 24827 sys_catalog.cc:455] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3212c0090954817961367ed0678e0f4. Latest consensus state: current_term: 1 leader_uuid: "b3212c0090954817961367ed0678e0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } } }
I20260924 16:16:20.467072 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3212c0090954817961367ed0678e0f4. Latest consensus state: current_term: 1 leader_uuid: "b3212c0090954817961367ed0678e0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } } }
I20260924 16:16:20.467211 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:16:20.467211 24827 sys_catalog.cc:458] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:16:20.471664 24792 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3212c0090954817961367ed0678e0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } } }
I20260924 16:16:20.471784 24792 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:20.471884 24792 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3212c0090954817961367ed0678e0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } } }
I20260924 16:16:20.471956 24792 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:20.473405 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3212c0090954817961367ed0678e0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } } }
I20260924 16:16:20.473519 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:16:20.474311 24827 sys_catalog.cc:455] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3212c0090954817961367ed0678e0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3212c0090954817961367ed0678e0f4" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 43889 } } peers { permanent_uuid: "a490df74eaea4ab08581c2a816242e6c" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 33429 } } peers { permanent_uuid: "591fd4e05e9c4307b8c5091ab652b584" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 45283 } } }
I20260924 16:16:20.474444 24827 sys_catalog.cc:458] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:16:20.474759 24844 mvcc.cc:205] Tried to move back new op lower bound from 7332931913554554880 to 7332931913449107456. Current Snapshot: MvccSnapshot[applied={T|T < 7332931913554554880}]
I20260924 16:16:20.475313 24846 mvcc.cc:205] Tried to move back new op lower bound from 7332931913554554880 to 7332931913449107456. Current Snapshot: MvccSnapshot[applied={T|T < 7332931913554554880}]
I20260924 16:16:20.483160 24828 catalog_manager.cc:1347] Generated new cluster ID: c31a27b3ccb94a4499825e7b02813472
I20260924 16:16:20.483251 24828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:20.484282 24845 mvcc.cc:205] Tried to move back new op lower bound from 7332931913554554880 to 7332931913449107456. Current Snapshot: MvccSnapshot[applied={T|T < 7332931913554554880}]
I20260924 16:16:20.509420 24828 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:20.510999 24828 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:20.527398 24828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3212c0090954817961367ed0678e0f4: Generated new TSK 0
I20260924 16:16:20.528386 24828 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:16:20.563915 24637 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:20.560376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.1.253.126:43889)
0924 16:16:20.560505 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.560510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.562529 (+ 2019us) server_negotiation.cc:407] Connection header received
0924 16:16:20.562938 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.562943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.563003 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.563061 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.563771 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58328: BlockingRecv error: recv got EOF from 127.0.0.1:58328 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 16:16:20.565541 24790 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:20.561234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.1.253.124:45283)
0924 16:16:20.561347 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.561351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.564323 (+ 2972us) server_negotiation.cc:407] Connection header received
0924 16:16:20.564676 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.564679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.564740 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.564792 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.565325 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57958: BlockingRecv error: recv got EOF from 127.0.0.1:57958 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 16:16:20.566841 24714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:20.561910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.1.253.125:33429)
0924 16:16:20.562025 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.562029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.565921 (+ 3892us) server_negotiation.cc:407] Connection header received
0924 16:16:20.566092 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.566095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.566143 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.566192 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.566724 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37638: BlockingRecv error: recv got EOF from 127.0.0.1:37638 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 16:16:20.571892 24714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:20.570275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37648 (local address 127.1.253.125:33429)
0924 16:16:20.570410 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.570415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.570886 (+ 471us) server_negotiation.cc:407] Connection header received
0924 16:16:20.571116 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.571120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.571213 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.571268 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.571787 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37648: BlockingRecv error: recv got EOF from 127.0.0.1:37648 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 16:16:20.577186 24714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:20.575683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.1.253.125:33429)
0924 16:16:20.575787 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.575791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.576289 (+ 498us) server_negotiation.cc:407] Connection header received
0924 16:16:20.576492 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.576495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.576545 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.576594 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.577082 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37654: BlockingRecv error: recv got EOF from 127.0.0.1:37654 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 16:16:20.583154 24714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:20.581523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.1.253.125:33429)
0924 16:16:20.581629 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.581633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.582149 (+ 516us) server_negotiation.cc:407] Connection header received
0924 16:16:20.582380 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.582384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.582450 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.582503 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.583034 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37666: BlockingRecv error: recv got EOF from 127.0.0.1:37666 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 16:16:20.591426 24714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:20.589661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.1.253.125:33429)
0924 16:16:20.589826 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.589831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.590416 (+ 585us) server_negotiation.cc:407] Connection header received
0924 16:16:20.590631 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.590634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.590693 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.590745 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.591291 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37676: BlockingRecv error: recv got EOF from 127.0.0.1:37676 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260924 16:16:20.604121 24714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:20.602148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37680 (local address 127.1.253.125:33429)
0924 16:16:20.602342 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.602348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.603005 (+ 657us) server_negotiation.cc:407] Connection header received
0924 16:16:20.603222 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.603226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.603292 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.603353 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.603974 (+ 621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37680: BlockingRecv error: recv got EOF from 127.0.0.1:37680 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 16:16:20.625200 24714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:20.623159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.1.253.125:33429)
0924 16:16:20.623289 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.623295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.624005 (+ 710us) server_negotiation.cc:407] Connection header received
0924 16:16:20.624301 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.624305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.624370 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.624441 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.625072 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37690: BlockingRecv error: recv got EOF from 127.0.0.1:37690 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 16:16:20.661701 24714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:20.660037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.1.253.125:33429)
0924 16:16:20.660180 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.660185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.660497 (+ 312us) server_negotiation.cc:407] Connection header received
0924 16:16:20.660745 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.660749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.660811 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.660871 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.661579 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37702: BlockingRecv error: recv got EOF from 127.0.0.1:37702 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 16:16:20.729283 24714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:20.728086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37706 (local address 127.1.253.125:33429)
0924 16:16:20.728232 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.728236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.728377 (+ 141us) server_negotiation.cc:407] Connection header received
0924 16:16:20.728518 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.728521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.728572 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.728659 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.729068 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37706: BlockingRecv error: recv got EOF from 127.0.0.1:37706 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 16:16:20.860962 24714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:20.859605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.1.253.125:33429)
0924 16:16:20.859779 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:20.859784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:20.859993 (+ 209us) server_negotiation.cc:407] Connection header received
0924 16:16:20.860190 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:20.860194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:20.860246 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:20.860339 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:20.860860 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37714: BlockingRecv error: recv got EOF from 127.0.0.1:37714 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 16:16:21.120458 24714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:21.119384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.1.253.125:33429)
0924 16:16:21.119537 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:21.119541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:21.119659 (+ 118us) server_negotiation.cc:407] Connection header received
0924 16:16:21.119806 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:21.119808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:21.119861 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:21.119943 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:21.120360 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37716: BlockingRecv error: recv got EOF from 127.0.0.1:37716 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:16:21.436893 24823 catalog_manager.cc:1259] Loaded cluster ID: c31a27b3ccb94a4499825e7b02813472
I20260924 16:16:21.437000 24823 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584: loading cluster ID for follower catalog manager: success
I20260924 16:16:21.438791 24823 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584: acquiring CA information for follower catalog manager: success
I20260924 16:16:21.439985 24823 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 591fd4e05e9c4307b8c5091ab652b584: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:16:21.449980 24842 catalog_manager.cc:1259] Loaded cluster ID: c31a27b3ccb94a4499825e7b02813472
I20260924 16:16:21.450064 24842 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c: loading cluster ID for follower catalog manager: success
I20260924 16:16:21.451924 24842 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c: acquiring CA information for follower catalog manager: success
I20260924 16:16:21.453136 24842 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a490df74eaea4ab08581c2a816242e6c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:16:21.638865 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:21.634691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.1.253.125:33429)
0924 16:16:21.634979 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:21.634983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:21.635055 (+ 72us) server_negotiation.cc:407] Connection header received
0924 16:16:21.635345 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:21.635348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:21.635546 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:21.635653 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:21.635903 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:21.636555 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:21.637374 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:21.637612 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:21.637691 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:21.637963 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:21.638439 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:16:21.638545 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:21.638618 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 16:16:21.638664 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260924 16:16:21.645442 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:21.642062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.1.253.124:45283)
0924 16:16:21.642321 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:21.642324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:21.642350 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:16:21.642513 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:21.642516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:21.642631 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:21.642731 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:21.642886 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:21.643403 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:21.644151 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:21.644361 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:21.644434 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:21.644691 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:21.645120 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:16:21.645212 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:21.645277 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 16:16:21.645323 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260924 16:16:21.652395 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:21.648228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.1.253.125:33429)
0924 16:16:21.648503 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:21.648510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:21.648918 (+ 408us) server_negotiation.cc:407] Connection header received
0924 16:16:21.649063 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:21.649067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:21.649129 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:21.649200 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:21.649433 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:21.650203 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:21.651265 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:21.651500 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:21.651618 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:21.651850 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:21.651933 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:16:21.652001 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:21.652172 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 16:16:21.652223 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260924 16:16:21.652815 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:21.648469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.1.253.124:45283)
0924 16:16:21.648593 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:21.648597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:21.648943 (+ 346us) server_negotiation.cc:407] Connection header received
0924 16:16:21.649210 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:21.649213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:21.649261 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:21.649344 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:21.649603 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:21.650120 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:21.651563 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:21.651874 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:21.652025 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:21.652292 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:21.652369 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:16:21.652418 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:21.652643 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 16:16:21.652703 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:16:21.653548 24930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:21.648249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.1.253.126:43889)
0924 16:16:21.648706 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:21.648710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:21.648954 (+ 244us) server_negotiation.cc:407] Connection header received
0924 16:16:21.649181 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:21.649184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:21.649355 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:21.649476 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:21.649723 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:21.650545 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:21.651676 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:21.652015 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:21.652112 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:21.652757 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:21.653138 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:16:21.653211 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:21.653329 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 16:16:21.653393 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":68,"threads_started":1}
I20260924 16:16:21.656664 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 24565
I20260924 16:16:21.664716 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 24640
I20260924 16:16:21.671746 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 24715
2026-09-24T16:16:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3154 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:16:21 dist-test-slave-6wvm krb5kdc[24936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:16:21 dist-test-slave-6wvm krb5kdc[24936](info): set up 2 sockets
Sep 24 16:16:21 dist-test-slave-6wvm krb5kdc[24936](info): commencing operation
krb5kdc: starting...
W20260924 16:16:22.734477 2037 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/test-user.keytab.
Sep 24 16:16:22 dist-test-slave-6wvm krb5kdc[24936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:16:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.253.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
Entry for principal kudu/127.1.253.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab.
I20260924 16:16:22.861297 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:42401
--webserver_interface=127.1.253.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:42401,127.1.253.125:45175,127.1.253.124:43251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:22.972949 24949 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:16:22.973316 24949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:22.973392 24949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:22.973480 24949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:22.977523 24949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:22.977628 24949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:22.977665 24949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:22.977706 24949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:22.977742 24949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:22.983719 24949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:42401,127.1.253.125:45175,127.1.253.124:43251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.126.keytab
--principal=kudu/127.1.253.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.126:42401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.253.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:22.984879 24949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:22.986047 24949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:22.994792 24955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:22.994962 24949 server_base.cc:1034] running on GCE node
W20260924 16:16:22.994822 24957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:22.994822 24954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:22.995831 24949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:22.997018 24949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:22.998209 24949 hybrid_clock.cc:648] HybridClock initialized: now 1790266582998184 us; error 28 us; skew 500 ppm
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:23.013259 24949 init.cc:377] Logged in from keytab as kudu/127.1.253.126@KRBTEST.COM (short username kudu)
I20260924 16:16:23.014515 24949 webserver.cc:460] Webserver started at http://127.1.253.126:37759/ using document root <none> and password file <none>
I20260924 16:16:23.015177 24949 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:23.015282 24949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:23.015552 24949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:23.017792 24949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/data/instance:
uuid: "c87f7dfdf200466a9420ba7b852f7274"
format_stamp: "Formatted at 2026-09-24 16:16:23 on dist-test-slave-6wvm"
server_key: "51a080ac8b33d79bf70cbb8ca3806d13"
server_key_iv: "9587be51be96538830ac9e7f2f484aff"
server_key_version: "encryptionkey@0"
I20260924 16:16:23.018471 24949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/wal/instance:
uuid: "c87f7dfdf200466a9420ba7b852f7274"
format_stamp: "Formatted at 2026-09-24 16:16:23 on dist-test-slave-6wvm"
server_key: "51a080ac8b33d79bf70cbb8ca3806d13"
server_key_iv: "9587be51be96538830ac9e7f2f484aff"
server_key_version: "encryptionkey@0"
I20260924 16:16:23.022522 24949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 16:16:23.025166 24964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:23.026691 24949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:23.026861 24949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/wal
uuid: "c87f7dfdf200466a9420ba7b852f7274"
format_stamp: "Formatted at 2026-09-24 16:16:23 on dist-test-slave-6wvm"
server_key: "51a080ac8b33d79bf70cbb8ca3806d13"
server_key_iv: "9587be51be96538830ac9e7f2f484aff"
server_key_version: "encryptionkey@0"
I20260924 16:16:23.026993 24949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:23.042908 24949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:23.046164 24949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:23.046446 24949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:23.053915 24949 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.126:42401
I20260924 16:16:23.054215 25018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.126:42401 every 8 connection(s)
I20260924 16:16:23.055047 24949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:23.057956 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:23.058802 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 24949
I20260924 16:16:23.058903 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-0/wal/instance
I20260924 16:16:23.059216 2037 external_mini_cluster.cc:1428] Setting key 7b8aaa86a119fdb1dd2691a689aa4739
I20260924 16:16:23.062067 25019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.253.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.125@KRBTEST.COM" created.
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:16:23.078150 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.064546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.1.253.126:42401)
0924 16:16:23.064999 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.065024 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.065316 (+ 292us) server_negotiation.cc:407] Connection header received
0924 16:16:23.066398 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.066404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.066624 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.066824 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.067723 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.068773 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.069962 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.070467 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.070689 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.073260 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.073266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.073276 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.073298 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.074977 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.075470 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.075475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.075477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.075585 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.075904 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.075907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.075908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.076299 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.076431 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.076822 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.077066 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":296,"thread_start_us":146,"threads_started":1}
I20260924 16:16:23.078150 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.064547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42401 (local address 127.0.0.1:57722)
0924 16:16:23.065160 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.065197 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:23.066052 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:23.066819 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:23.066824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:23.067275 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:23.067556 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.067583 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.068894 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.068899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.069827 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.069841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.070630 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.070634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.070788 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.071418 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:23.071435 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:23.073113 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:23.075105 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.075122 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.075132 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.075358 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.075690 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.075693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.075695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.075807 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.076416 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:23.076422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:23.076562 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 16:16:23.076744 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 16:16:23.077125 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":137,"spinlock_wait_cycles":768,"thread_start_us":158,"threads_started":1}
I20260924 16:16:23.080809 25019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:16:23.081514 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.081228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:45175 (local address 127.0.0.1:47026)
0924 16:16:23.081352 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.081449 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:45175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:16:23.081784 24968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.253.125:45175: connect: Connection refused (error 111)
I20260924 16:16:23.082779 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.082520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:45175 (local address 127.0.0.1:47038)
0924 16:16:23.082645 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.082691 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:45175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:16:23.083105 25019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.125:45175: Network error: Client connection negotiation failed: client connection to 127.1.253.125:45175: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.125.keytab.
Entry for principal kudu/127.1.253.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.125.keytab.
I20260924 16:16:23.085417 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.125:45175
--webserver_interface=127.1.253.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.125.keytab
--principal=kudu/127.1.253.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:42401,127.1.253.125:45175,127.1.253.124:43251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:16:23.132326 25019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } attempt: 1
I20260924 16:16:23.133246 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.132939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:45175 (local address 127.0.0.1:47052)
0924 16:16:23.133103 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.133168 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:45175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:16:23.134024 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.133792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:45175 (local address 127.0.0.1:47064)
0924 16:16:23.133912 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.133965 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:45175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:16:23.134140 25019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.125:45175: Network error: Client connection negotiation failed: client connection to 127.1.253.125:45175: connect: Connection refused (error 111)
W20260924 16:16:23.197299 25026 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:16:23.197644 25026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:23.197719 25026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:23.197803 25026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:23.201843 25026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:23.201938 25026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:23.201975 25026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:23.202011 25026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:23.202050 25026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:23.202288 25019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } attempt: 2
I20260924 16:16:23.203213 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.202906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:45175 (local address 127.0.0.1:47066)
0924 16:16:23.203073 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.203136 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:45175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:16:23.204015 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.203756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:45175 (local address 127.0.0.1:47074)
0924 16:16:23.203891 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.203947 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.125:45175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 16:16:23.204155 25019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.125:45175: Network error: Client connection negotiation failed: client connection to 127.1.253.125:45175: connect: Connection refused (error 111)
I20260924 16:16:23.208038 25026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:42401,127.1.253.125:45175,127.1.253.124:43251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.125.keytab
--principal=kudu/127.1.253.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.125:45175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.253.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:23.209151 25026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:23.210345 25026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:23.218946 25034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:23.218956 25032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:23.219013 25026 server_base.cc:1034] running on GCE node
W20260924 16:16:23.218962 25031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:23.219719 25026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:23.220913 25026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:23.222095 25026 hybrid_clock.cc:648] HybridClock initialized: now 1790266583222080 us; error 34 us; skew 500 ppm
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:23.224937 25026 init.cc:377] Logged in from keytab as kudu/127.1.253.125@KRBTEST.COM (short username kudu)
I20260924 16:16:23.225863 25026 webserver.cc:460] Webserver started at http://127.1.253.125:45673/ using document root <none> and password file <none>
I20260924 16:16:23.226522 25026 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:23.226615 25026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:23.226859 25026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:23.228960 25026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/data/instance:
uuid: "a315f2ebd9f6469abb0b763145bd6617"
format_stamp: "Formatted at 2026-09-24 16:16:23 on dist-test-slave-6wvm"
server_key: "eb961ac372bf064abb361345eacdce81"
server_key_iv: "f266452ccf11be4d2bde808730ea0795"
server_key_version: "encryptionkey@0"
I20260924 16:16:23.229542 25026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/wal/instance:
uuid: "a315f2ebd9f6469abb0b763145bd6617"
format_stamp: "Formatted at 2026-09-24 16:16:23 on dist-test-slave-6wvm"
server_key: "eb961ac372bf064abb361345eacdce81"
server_key_iv: "f266452ccf11be4d2bde808730ea0795"
server_key_version: "encryptionkey@0"
I20260924 16:16:23.233683 25026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 16:16:23.236241 25041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:23.237447 25026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 16:16:23.237593 25026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/wal
uuid: "a315f2ebd9f6469abb0b763145bd6617"
format_stamp: "Formatted at 2026-09-24 16:16:23 on dist-test-slave-6wvm"
server_key: "eb961ac372bf064abb361345eacdce81"
server_key_iv: "f266452ccf11be4d2bde808730ea0795"
server_key_version: "encryptionkey@0"
I20260924 16:16:23.237707 25026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:23.257311 25026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:23.260751 25026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:23.260962 25026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:23.268141 25026 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.125:45175
I20260924 16:16:23.268406 25093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.125:45175 every 8 connection(s)
I20260924 16:16:23.269158 25026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/data/info.pb
I20260924 16:16:23.271759 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 25026
I20260924 16:16:23.271900 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-1/wal/instance
I20260924 16:16:23.272048 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:23.272212 2037 external_mini_cluster.cc:1428] Setting key c1bc30e958952c60911c396fc0e7e4ab
I20260924 16:16:23.276460 25094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.253.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.253.124@KRBTEST.COM" created.
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.125@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:16:23.291239 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.278435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57736 (local address 127.1.253.126:42401)
0924 16:16:23.278610 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.278625 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.279471 (+ 846us) server_negotiation.cc:407] Connection header received
0924 16:16:23.280462 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.280469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.280538 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.280649 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.281792 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.282514 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.283882 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.284121 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.284235 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.287129 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.287133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.287137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.287172 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.288771 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.289426 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.289430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.289431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.289518 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.289977 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.289980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.289981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.290155 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.290231 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.291000 (+ 769us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.291113 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:16:23.292352 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.278807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42401 (local address 127.0.0.1:57736)
0924 16:16:23.279286 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.279348 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:23.280227 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:23.280815 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:23.280821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:23.281160 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:23.281649 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.281662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.282680 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.282684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.283727 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.283738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.284294 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.284298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.284476 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.285088 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:23.285104 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:23.286919 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:23.288904 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.288912 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.288930 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.289311 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.289640 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.289643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.289645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.289848 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.290302 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:23.290310 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:23.290671 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 16:16:23.290946 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 16:16:23.291316 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":140,"threads_started":1}
I20260924 16:16:23.293939 25094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:16:23.295307 25019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.253.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.124.keytab.
Entry for principal kudu/127.1.253.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.124.keytab.
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.125@KRBTEST.COM for kudu/127.1.253.125@KRBTEST.COM
I20260924 16:16:23.300889 2037 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pu02A/build/debug/bin/kudu
/tmp/dist-test-task6Pu02A/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.124:43251
--webserver_interface=127.1.253.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.124.keytab
--principal=kudu/127.1.253.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.253.84:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.253.126:42401,127.1.253.125:45175,127.1.253.124:43251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.125@KRBTEST.COM
I20260924 16:16:23.305874 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.294711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:45175 (local address 127.0.0.1:47090)
0924 16:16:23.294821 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.294833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:23.294898 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:23.295615 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:23.295618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:23.295791 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:23.295884 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.295894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.297178 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.297181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.297780 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.297828 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.298682 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.298685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.298758 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.299282 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:23.299296 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:23.300734 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:23.303792 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.303796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.303799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.304073 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.304424 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.304430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.304435 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.304540 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.305170 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:23.305175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:23.305276 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 16:16:23.305440 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 16:16:23.305630 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":1}
I20260924 16:16:23.306098 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.295737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:45175 (local address 127.0.0.1:47096)
0924 16:16:23.295862 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.295876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:23.295992 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:23.296528 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:23.296531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:23.296685 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:23.296819 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.296829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.297622 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.297629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.299140 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.299153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.299767 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.299772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.299914 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.300477 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:23.300489 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:23.301930 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:23.304555 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.304559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.304561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.304758 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.305066 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.305069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.305071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.305155 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.305627 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:23.305630 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:23.305734 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 16:16:23.305819 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 16:16:23.305952 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 16:16:23.305896 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.294840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.1.253.125:45175)
0924 16:16:23.295131 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.295141 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.295160 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:16:23.295196 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.295200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.295368 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.295644 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.296003 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.297054 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.297945 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.298553 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.298681 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.300907 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.300921 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.300939 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.300969 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.303663 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.304206 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.304215 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.304217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.304312 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.304656 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.304661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.304666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.304992 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.305130 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.305536 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.305694 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":113,"threads_started":1}
I20260924 16:16:23.306814 25100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.295932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.1.253.125:45175)
0924 16:16:23.296303 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.296312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.296325 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:16:23.296356 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.296359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.296406 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.296490 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.296942 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.297516 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.299272 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.299645 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.299776 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.302090 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.302093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.302095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.302123 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.304441 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.304861 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.304866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.304868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.304969 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.305250 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.305254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.305256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.305508 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.306458 (+ 950us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.306546 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.306660 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"spinlock_wait_cycles":7296,"thread_start_us":128,"threads_started":1}
I20260924 16:16:23.308471 25019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:16:23.308630 25094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:16:23.309340 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.309021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46336)
0924 16:16:23.309179 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.309248 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 16:16:23.309885 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.309021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46346)
0924 16:16:23.309712 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.309803 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596}
I20260924 16:16:23.310127 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.309842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46350)
0924 16:16:23.309992 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.310065 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 16:16:23.310200 25042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
W20260924 16:16:23.311023 25019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:43251: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
I20260924 16:16:23.311177 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.310883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46366)
0924 16:16:23.311019 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.311099 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:16:23.311301 25094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:43251: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
I20260924 16:16:23.342191 25019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } attempt: 1
I20260924 16:16:23.343127 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.342798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46376)
0924 16:16:23.342961 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.343040 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 16:16:23.344058 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.343648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46380)
0924 16:16:23.343933 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.343997 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260924 16:16:23.344205 25019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:43251: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
I20260924 16:16:23.360564 25094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } attempt: 1
I20260924 16:16:23.361402 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.361113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46396)
0924 16:16:23.361264 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.361323 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 16:16:23.362143 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.361909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46412)
0924 16:16:23.362043 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.362088 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:16:23.362290 25094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:43251: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
W20260924 16:16:23.417680 25101 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:16:23.417949 25101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:23.418004 25101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:23.418078 25101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 16:16:23.419425 25019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } attempt: 2
I20260924 16:16:23.420343 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.420022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46414)
0924 16:16:23.420192 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.420264 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 16:16:23.421154 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.420927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46430)
0924 16:16:23.421043 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.421090 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 16:16:23.421310 25019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:43251: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
W20260924 16:16:23.422348 25101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:23.422395 25101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:23.422415 25101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:23.422463 25101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:23.422498 25101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:23.427757 25101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.253.84:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.253.126:42401,127.1.253.125:45175,127.1.253.124:43251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/krb5kdc/kudu_127.1.253.124.keytab
--principal=kudu/127.1.253.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.253.124:43251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.253.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 15:14:23 UTC on ccycloud.quasar-qkwurp.root.comops.site
build id 4670390
I20260924 16:16:23.428785 25101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:23.429813 25101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:16:23.430482 25094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } attempt: 2
I20260924 16:16:23.431228 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.430968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46432)
0924 16:16:23.431105 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.431161 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 16:16:23.432096 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.431903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46436)
0924 16:16:23.432004 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.432048 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 16:16:23.432242 25094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.253.124:43251: Network error: Client connection negotiation failed: client connection to 127.1.253.124:43251: connect: Connection refused (error 111)
W20260924 16:16:23.438747 25110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:23.438750 25107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:23.438777 25108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:23.439102 25101 server_base.cc:1034] running on GCE node
I20260924 16:16:23.439653 25101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:23.440807 25101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:23.441967 25101 hybrid_clock.cc:648] HybridClock initialized: now 1790266583441950 us; error 28 us; skew 500 ppm
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:23.444795 25101 init.cc:377] Logged in from keytab as kudu/127.1.253.124@KRBTEST.COM (short username kudu)
I20260924 16:16:23.445745 25101 webserver.cc:460] Webserver started at http://127.1.253.124:38611/ using document root <none> and password file <none>
I20260924 16:16:23.446380 25101 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:23.446472 25101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:23.446720 25101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:23.448770 25101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/data/instance:
uuid: "94fda696ccbd4da895a74a80a6ed8a82"
format_stamp: "Formatted at 2026-09-24 16:16:23 on dist-test-slave-6wvm"
server_key: "89e5adc5f41d6cf6f2d55f1828f196d0"
server_key_iv: "570f097e808390cac7f85f4a3b695d5b"
server_key_version: "encryptionkey@0"
I20260924 16:16:23.449340 25101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/wal/instance:
uuid: "94fda696ccbd4da895a74a80a6ed8a82"
format_stamp: "Formatted at 2026-09-24 16:16:23 on dist-test-slave-6wvm"
server_key: "89e5adc5f41d6cf6f2d55f1828f196d0"
server_key_iv: "570f097e808390cac7f85f4a3b695d5b"
server_key_version: "encryptionkey@0"
I20260924 16:16:23.453004 25101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 16:16:23.455546 25118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:23.456730 25101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 16:16:23.456882 25101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/data,/tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/wal
uuid: "94fda696ccbd4da895a74a80a6ed8a82"
format_stamp: "Formatted at 2026-09-24 16:16:23 on dist-test-slave-6wvm"
server_key: "89e5adc5f41d6cf6f2d55f1828f196d0"
server_key_iv: "570f097e808390cac7f85f4a3b695d5b"
server_key_version: "encryptionkey@0"
I20260924 16:16:23.456995 25101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:23.475622 25101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:23.479105 25101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:23.479321 25101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:23.486825 25101 rpc_server.cc:307] RPC server started. Bound to: 127.1.253.124:43251
I20260924 16:16:23.487123 25172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.253.124:43251 every 8 connection(s)
I20260924 16:16:23.487893 25101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/data/info.pb
I20260924 16:16:23.488713 2037 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pu02A/build/debug/bin/kudu as pid 25101
I20260924 16:16:23.488874 2037 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pu02A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266297063660-2037-0/minicluster-data/master-2/wal/instance
I20260924 16:16:23.489244 2037 external_mini_cluster.cc:1428] Setting key a3cf87efde3746dcd8ff753202dbbcfa
I20260924 16:16:23.491715 25173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:23.496125 25173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.124@KRBTEST.COM
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.124@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
I20260924 16:16:23.505010 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.490863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.1.253.124:43251)
0924 16:16:23.491370 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.491388 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.491431 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:16:23.492205 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.492226 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.492596 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.492982 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.493822 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.494951 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.495870 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.496351 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.496521 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.499593 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.499599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.499609 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.499637 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.501431 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.501913 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.501918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.501923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.502043 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.502433 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.502437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.502442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.502997 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.503243 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.503617 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.503953 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":149,"threads_started":1}
I20260924 16:16:23.508330 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.498172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.1.253.126:42401)
0924 16:16:23.498366 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.498371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.498763 (+ 392us) server_negotiation.cc:407] Connection header received
0924 16:16:23.499055 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.499060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.499126 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.499231 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.500016 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.500525 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.501480 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.501720 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.501844 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.504414 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.504418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.504422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.504454 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.506142 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.506745 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.506749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.506750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.506833 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.507137 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.507141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.507142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.507409 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.507520 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.508007 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.508138 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:16:23.508988 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.498087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.126:42401 (local address 127.0.0.1:57748)
0924 16:16:23.498614 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.498660 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:23.498829 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:23.499321 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:23.499326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:23.499697 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:23.499836 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.499843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.500679 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.500683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.501326 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.501336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.501859 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.501863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.501995 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.502518 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:23.502531 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:23.504260 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:23.506318 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.506328 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.506334 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.506613 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.506940 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.506943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.506945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.507032 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.507544 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:23.507549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:23.507726 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 16:16:23.508490 (+ 764us) client_negotiation.cc:240] Negotiation successful
0924 16:16:23.508711 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":208,"threads_started":1}
I20260924 16:16:23.510303 25173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.126@KRBTEST.COM
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.124@KRBTEST.COM for kudu/127.1.253.125@KRBTEST.COM
I20260924 16:16:23.518790 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.507954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.1.253.126:42401)
0924 16:16:23.508388 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.508400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.508414 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:16:23.508462 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.508464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.508517 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.508602 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.509510 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.510178 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.511263 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.511601 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.511723 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.514483 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.514486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.514488 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.514509 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.516351 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.516888 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.516895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.516939 (+ 44us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.517056 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.517664 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.517668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.517670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.517897 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.518166 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.518435 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.518572 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":172,"threads_started":1}
I20260924 16:16:23.519840 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.510883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.125:45175 (local address 127.0.0.1:47098)
0924 16:16:23.511020 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.511038 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:23.511125 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:23.511476 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:23.511479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:23.511688 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:23.511825 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.511833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.512973 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.512976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.513687 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.513698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.514324 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.514328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.514450 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.514929 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:23.514948 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:23.516478 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:23.518182 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.518187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.518189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.518461 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.518820 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.518830 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.518832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.518937 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.519363 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:23.519365 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:23.519438 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 16:16:23.519591 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 16:16:23.519726 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 16:16:23.520051 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.511079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.1.253.125:45175)
0924 16:16:23.511255 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.511260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.511274 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:16:23.511323 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.511325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.511366 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.511427 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.512000 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.512837 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.513813 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.514156 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.514293 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.516650 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.516653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.516655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.516681 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.518047 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.518577 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.518584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.518586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.518701 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.519055 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.519058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.519060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.519243 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.519301 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.519682 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.519826 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:16:23.520475 25019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } attempt: 3
I20260924 16:16:23.521157 25173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:16:23.523428 25094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } attempt: 3
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.253.125@KRBTEST.COM
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.126@KRBTEST.COM for kudu/127.1.253.124@KRBTEST.COM
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.125@KRBTEST.COM for kudu/127.1.253.124@KRBTEST.COM
Sep 24 16:16:23 dist-test-slave-6wvm krb5kdc[24936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.253.124@KRBTEST.COM for kudu/127.1.253.124@KRBTEST.COM
I20260924 16:16:23.532081 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.520402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.1.253.125:45175)
0924 16:16:23.520514 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.520519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.520541 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:16:23.520607 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.520612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.520678 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.520791 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.521738 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.522405 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.523496 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.523865 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.523983 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.527441 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.527447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.527451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.527483 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.529480 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.529951 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.529957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.529959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.530039 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.530460 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.530464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.530465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.530716 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.531664 (+ 948us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.531751 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.531876 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 16:16:23.533133 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.520897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46456)
0924 16:16:23.521142 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.521153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:23.521219 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:23.521641 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:23.521648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:23.521864 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:23.522002 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.522014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.523753 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.523760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.524497 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.524507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.525142 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.525146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.525244 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.525779 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:23.525793 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:23.527867 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:23.530853 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.530860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.530865 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.531195 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.531590 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.531594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.531597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.531777 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.532522 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:23.532526 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:23.532654 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 16:16:23.532757 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 16:16:23.532917 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260924 16:16:23.533736 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.520949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.1.253.124:43251)
0924 16:16:23.521383 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.521387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.521397 (+ 10us) server_negotiation.cc:407] Connection header received
0924 16:16:23.521424 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.521426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.521457 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.521590 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.522137 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.523611 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.524651 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.525003 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.525455 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.528083 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.528087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.528091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.528123 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.530705 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.531345 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.531350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.531356 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.531457 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.531947 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.531951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.531954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.532380 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.533336 (+ 956us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.533436 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.533567 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":329}
I20260924 16:16:23.535039 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.521683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46464)
0924 16:16:23.522603 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.522618 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:23.522707 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:23.522970 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:23.522973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:23.523152 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:23.523232 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.523238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.525070 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.525076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.526573 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.526586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.527339 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.527343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.527653 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.529039 (+ 1386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:23.529057 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:23.531013 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:23.533584 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.533590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.533592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.533807 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.534094 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.534098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.534103 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.534182 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.534602 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:23.534608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:23.534697 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 16:16:23.534776 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 16:16:23.534896 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":146}
I20260924 16:16:23.535562 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.521908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.1.253.124:43251)
0924 16:16:23.522486 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.522496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.522689 (+ 193us) server_negotiation.cc:407] Connection header received
0924 16:16:23.522787 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.522790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.522839 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.522935 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.523359 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.524941 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.526715 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.527200 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.529736 (+ 2536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.531154 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.531157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.531160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.531182 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.533468 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.533918 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.533921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.533923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.534000 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.534299 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.534302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.534303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.534486 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.535174 (+ 688us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.535266 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.535380 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":136,"threads_started":1}
I20260924 16:16:23.535969 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.523801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.253.124:43251 (local address 127.0.0.1:46472)
0924 16:16:23.523953 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:23.523966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:23.524048 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:23.524751 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:23.524754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:23.524923 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:23.525015 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.525021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.526217 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.526222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.526944 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:23.526955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.527786 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.527790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:23.527877 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.528346 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:23.528367 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:23.529795 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:23.533058 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.533061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.533064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.533345 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.534427 (+ 1082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:23.534435 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:23.534437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:23.534552 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:23.535450 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:23.535454 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:23.535568 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 16:16:23.535659 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:16:23.535808 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 16:16:23.536249 2037 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:16:23.536649 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.524232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.1.253.124:43251)
0924 16:16:23.524519 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.524527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.524541 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:16:23.524575 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.524577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.524623 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.525133 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.525318 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.526079 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.527121 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.527646 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.527785 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.529920 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.529926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.529933 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.529963 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.532918 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.534169 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.534172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.534174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.534290 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.534684 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.534687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.534688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.535308 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.536269 (+ 961us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.536358 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.536484 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":225,"thread_start_us":138,"threads_started":1}
I20260924 16:16:23.537624 25019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274: Bootstrap starting.
I20260924 16:16:23.537663 24992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c87f7dfdf200466a9420ba7b852f7274"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57756
I20260924 16:16:23.539726 25094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617: Bootstrap starting.
I20260924 16:16:23.540843 25019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:23.541697 25019 log.cc:824] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:23.542716 25094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:23.543430 25094 log.cc:824] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:23.543999 25019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274: No bootstrap required, opened a new log
I20260924 16:16:23.544562 25173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82: Bootstrap starting.
I20260924 16:16:23.545409 25094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617: No bootstrap required, opened a new log
I20260924 16:16:23.547081 25173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:23.547179 25019 raft_consensus.cc:348] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } }
I20260924 16:16:23.547423 25019 raft_consensus.cc:729] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87f7dfdf200466a9420ba7b852f7274, State: Initialized, Role: FOLLOWER
I20260924 16:16:23.547761 25019 consensus_queue.cc:260] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } }
I20260924 16:16:23.547911 25173 log.cc:824] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:23.548496 25183 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } } }
I20260924 16:16:23.548830 25183 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:16:23.549031 25019 sys_catalog.cc:564] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:23.549137 24992 raft_consensus.cc:482] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 16:16:23.549256 24992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:23.550103 25094 raft_consensus.cc:348] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } }
I20260924 16:16:23.550410 25094 raft_consensus.cc:729] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a315f2ebd9f6469abb0b763145bd6617, State: Initialized, Role: FOLLOWER
I20260924 16:16:23.550755 24992 raft_consensus.cc:504] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } }
I20260924 16:16:23.550778 25094 consensus_queue.cc:260] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } }
I20260924 16:16:23.552343 25094 sys_catalog.cc:564] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:23.553414 25067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c87f7dfdf200466a9420ba7b852f7274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a315f2ebd9f6469abb0b763145bd6617"
I20260924 16:16:23.553551 25173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82: No bootstrap required, opened a new log
I20260924 16:16:23.553685 25067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:23.555287 25067 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87f7dfdf200466a9420ba7b852f7274 in term 1.
I20260924 16:16:23.555428 25193 sys_catalog.cc:455] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } } }
I20260924 16:16:23.555557 25193 sys_catalog.cc:458] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:16:23.551841 24992 leader_election.cc:288] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [CANDIDATE]: Term 1 election: Requested vote from peers a315f2ebd9f6469abb0b763145bd6617 (127.1.253.125:45175), 94fda696ccbd4da895a74a80a6ed8a82 (127.1.253.124:43251)
I20260924 16:16:23.556152 24968 leader_election.cc:302] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a315f2ebd9f6469abb0b763145bd6617, c87f7dfdf200466a9420ba7b852f7274; no voters:
I20260924 16:16:23.558228 25173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } }
I20260924 16:16:23.558498 25173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fda696ccbd4da895a74a80a6ed8a82, State: Initialized, Role: FOLLOWER
I20260924 16:16:23.558815 25183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:23.558837 25173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } }
I20260924 16:16:23.559221 25183 raft_consensus.cc:686] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [term 1 LEADER]: Becoming Leader. State: Replica: c87f7dfdf200466a9420ba7b852f7274, State: Running, Role: LEADER
I20260924 16:16:23.559831 25183 consensus_queue.cc:237] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } }
I20260924 16:16:23.560256 25173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:23.560308 25145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c87f7dfdf200466a9420ba7b852f7274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "94fda696ccbd4da895a74a80a6ed8a82"
I20260924 16:16:23.560559 25145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [term 0 FOLLOWER]: Advancing to term 1
W20260924 16:16:23.554838 25197 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:16:23.561677 25197 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:16:23.562107 25145 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87f7dfdf200466a9420ba7b852f7274 in term 1.
W20260924 16:16:23.565702 25227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:16:23.565796 25227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:16:23.566296 24949 master_runner.cc:263] No leader master found from master c87f7dfdf200466a9420ba7b852f7274
I20260924 16:16:23.566380 24949 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:16:23.568774 25183 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87f7dfdf200466a9420ba7b852f7274. Latest consensus state: current_term: 1 leader_uuid: "c87f7dfdf200466a9420ba7b852f7274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } } }
I20260924 16:16:23.568930 25183 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:23.576210 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.562177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.1.253.126:42401)
0924 16:16:23.562344 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.562350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.566125 (+ 3775us) server_negotiation.cc:407] Connection header received
0924 16:16:23.566351 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.566355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.566414 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.566469 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.567643 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.568456 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.569692 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.570076 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.570160 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.572247 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.572251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.572254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.572282 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.574157 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.574698 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.574702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.574704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.574795 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.575130 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.575133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.575135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.575367 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.575515 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.575896 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.576038 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 16:16:23.579175 25214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:16:23.579279 25214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:16:23.579525 25215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } } }
I20260924 16:16:23.579675 25215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:16:23.585466 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.563541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.1.253.125:45175)
0924 16:16:23.563780 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.563785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.570918 (+ 7133us) server_negotiation.cc:407] Connection header received
0924 16:16:23.572634 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.572637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.572699 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.572754 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.577632 (+ 4878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.578461 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.579618 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.579946 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.580038 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.581779 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.581783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.581786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.581812 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.583552 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.584058 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.584063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.584067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.584154 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.584472 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.584475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.584477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.584697 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.584816 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.585189 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.585305 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260924 16:16:23.590728 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:23.563473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.1.253.124:43251)
0924 16:16:23.563684 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.563691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.574635 (+ 10944us) server_negotiation.cc:407] Connection header received
0924 16:16:23.574685 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.574688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.574744 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.574822 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.581622 (+ 6800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.583135 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.584529 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:23.584957 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:23.585043 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:23.587067 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:23.587071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:23.587074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:23.587101 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:23.588821 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.589350 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.589354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.589356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.589445 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:23.589763 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:23.589766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:23.589768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:23.589984 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:23.590121 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:23.590440 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 16:16:23.590565 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260924 16:16:23.594669 25233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:23.595947 25233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:23.607812 25145 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [term 1 FOLLOWER]: Refusing update from remote peer c87f7dfdf200466a9420ba7b852f7274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:16:23.607821 25067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [term 1 FOLLOWER]: Refusing update from remote peer c87f7dfdf200466a9420ba7b852f7274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:16:23.608724 25216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:16:23.609892 25216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:16:23.617895 25193 sys_catalog.cc:455] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87f7dfdf200466a9420ba7b852f7274. Latest consensus state: current_term: 1 leader_uuid: "c87f7dfdf200466a9420ba7b852f7274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } } }
I20260924 16:16:23.618069 25193 sys_catalog.cc:458] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:16:23.620967 25215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87f7dfdf200466a9420ba7b852f7274. Latest consensus state: current_term: 1 leader_uuid: "c87f7dfdf200466a9420ba7b852f7274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } } }
I20260924 16:16:23.621114 25215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:16:23.631295 25216 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c87f7dfdf200466a9420ba7b852f7274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } } }
I20260924 16:16:23.631449 25216 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:23.632427 25193 sys_catalog.cc:455] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c87f7dfdf200466a9420ba7b852f7274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } } }
I20260924 16:16:23.632561 25193 sys_catalog.cc:458] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:16:23.635318 25237 mvcc.cc:205] Tried to move back new op lower bound from 7332931926438453248 to 7332931926292312064. Current Snapshot: MvccSnapshot[applied={T|T < 7332931926438453248}]
I20260924 16:16:23.636233 25183 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c87f7dfdf200466a9420ba7b852f7274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } } }
I20260924 16:16:23.636372 25183 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:23.639410 25215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c87f7dfdf200466a9420ba7b852f7274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87f7dfdf200466a9420ba7b852f7274" member_type: VOTER last_known_addr { host: "127.1.253.126" port: 42401 } } peers { permanent_uuid: "a315f2ebd9f6469abb0b763145bd6617" member_type: VOTER last_known_addr { host: "127.1.253.125" port: 45175 } } peers { permanent_uuid: "94fda696ccbd4da895a74a80a6ed8a82" member_type: VOTER last_known_addr { host: "127.1.253.124" port: 43251 } } }
I20260924 16:16:23.639550 25215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:16:23.641932 25236 mvcc.cc:205] Tried to move back new op lower bound from 7332931926438453248 to 7332931926292312064. Current Snapshot: MvccSnapshot[applied={T|T < 7332931926438453248}]
I20260924 16:16:23.646795 25233 catalog_manager.cc:1347] Generated new cluster ID: eeee6ae1b71a4d258d97620be3127dc0
I20260924 16:16:23.646875 25233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:23.666013 25233 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:23.667465 25233 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:23.684885 25233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c87f7dfdf200466a9420ba7b852f7274: Generated new TSK 0
I20260924 16:16:23.685896 25233 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:16:23.727172 25180 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:23.723645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.1.253.124:43251)
0924 16:16:23.723801 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.723807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.725954 (+ 2147us) server_negotiation.cc:407] Connection header received
0924 16:16:23.726176 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.726180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.726232 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.726317 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.727026 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46488: BlockingRecv error: recv got EOF from 127.0.0.1:46488 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 16:16:23.728623 25178 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:23.724144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.1.253.126:42401)
0924 16:16:23.724289 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.724294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.727673 (+ 3379us) server_negotiation.cc:407] Connection header received
0924 16:16:23.727859 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.727862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.727971 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.728027 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.728510 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57780: BlockingRecv error: recv got EOF from 127.0.0.1:57780 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 16:16:23.729990 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:23.725293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.1.253.125:45175)
0924 16:16:23.725402 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.725406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.729094 (+ 3688us) server_negotiation.cc:407] Connection header received
0924 16:16:23.729264 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.729268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.729318 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.729368 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.729867 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47138: BlockingRecv error: recv got EOF from 127.0.0.1:47138 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 16:16:23.734700 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:23.733097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.1.253.125:45175)
0924 16:16:23.733208 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.733212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.733662 (+ 450us) server_negotiation.cc:407] Connection header received
0924 16:16:23.733872 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.733876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.733967 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.734025 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.734595 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47150: BlockingRecv error: recv got EOF from 127.0.0.1:47150 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 16:16:23.739988 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:23.738476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.1.253.125:45175)
0924 16:16:23.738587 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.738591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.739090 (+ 499us) server_negotiation.cc:407] Connection header received
0924 16:16:23.739279 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.739282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.739333 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.739382 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.739875 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47156: BlockingRecv error: recv got EOF from 127.0.0.1:47156 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 16:16:23.751545 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:23.749636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.1.253.125:45175)
0924 16:16:23.749759 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.749763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.750471 (+ 708us) server_negotiation.cc:407] Connection header received
0924 16:16:23.750687 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.750691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.750748 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.750802 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.751422 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47172: BlockingRecv error: recv got EOF from 127.0.0.1:47172 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 16:16:23.760020 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:23.758114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.1.253.125:45175)
0924 16:16:23.758288 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.758293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.758949 (+ 656us) server_negotiation.cc:407] Connection header received
0924 16:16:23.759177 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.759181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.759239 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.759296 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.759889 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47184: BlockingRecv error: recv got EOF from 127.0.0.1:47184 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 16:16:23.772709 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:23.770797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.1.253.125:45175)
0924 16:16:23.770944 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.770949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.771631 (+ 682us) server_negotiation.cc:407] Connection header received
0924 16:16:23.771865 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.771869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.771940 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.772002 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.772587 (+ 585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47190: BlockingRecv error: recv got EOF from 127.0.0.1:47190 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 16:16:23.793015 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:23.791574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.1.253.125:45175)
0924 16:16:23.791716 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.791722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.792015 (+ 293us) server_negotiation.cc:407] Connection header received
0924 16:16:23.792254 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.792258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.792321 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.792414 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.792892 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47194: BlockingRecv error: recv got EOF from 127.0.0.1:47194 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 16:16:23.828380 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:23.827267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.1.253.125:45175)
0924 16:16:23.827447 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.827451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.827519 (+ 68us) server_negotiation.cc:407] Connection header received
0924 16:16:23.827654 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.827657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.827711 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.827786 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.828251 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47204: BlockingRecv error: recv got EOF from 127.0.0.1:47204 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 16:16:23.899081 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:23.897419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.1.253.125:45175)
0924 16:16:23.897605 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:23.897612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:23.897952 (+ 340us) server_negotiation.cc:407] Connection header received
0924 16:16:23.898166 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:23.898170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:23.898273 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:23.898371 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:23.898946 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47208: BlockingRecv error: recv got EOF from 127.0.0.1:47208 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 16:16:24.030653 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:24.029255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.1.253.125:45175)
0924 16:16:24.029417 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:24.029423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:24.029631 (+ 208us) server_negotiation.cc:407] Connection header received
0924 16:16:24.029838 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:24.029841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:24.029893 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:24.030026 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:24.030528 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47218: BlockingRecv error: recv got EOF from 127.0.0.1:47218 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 16:16:24.290062 25099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:16:24.288936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.1.253.125:45175)
0924 16:16:24.289105 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:24.289109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:24.289208 (+ 99us) server_negotiation.cc:407] Connection header received
0924 16:16:24.289336 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:24.289338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:24.289387 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:24.289478 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:24.289954 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47226: BlockingRecv error: recv got EOF from 127.0.0.1:47226 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:16:24.567209 25227 catalog_manager.cc:1259] Loaded cluster ID: eeee6ae1b71a4d258d97620be3127dc0
I20260924 16:16:24.567297 25227 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82: loading cluster ID for follower catalog manager: success
I20260924 16:16:24.569033 25227 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82: acquiring CA information for follower catalog manager: success
I20260924 16:16:24.570338 25227 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94fda696ccbd4da895a74a80a6ed8a82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:16:24.580518 25214 catalog_manager.cc:1259] Loaded cluster ID: eeee6ae1b71a4d258d97620be3127dc0
I20260924 16:16:24.580591 25214 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617: loading cluster ID for follower catalog manager: success
I20260924 16:16:24.582177 25214 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617: acquiring CA information for follower catalog manager: success
I20260924 16:16:24.583252 25214 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a315f2ebd9f6469abb0b763145bd6617: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:16:24.808578 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:24.804806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.1.253.125:45175)
0924 16:16:24.805098 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:24.805102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:24.805119 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:16:24.805178 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:24.805180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:24.805308 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:24.805410 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:24.805696 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.806296 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.807137 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.807368 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.807459 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:24.807755 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:24.808234 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:16:24.808327 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:24.808386 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 16:16:24.808440 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260924 16:16:24.815475 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:24.811875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.1.253.124:43251)
0924 16:16:24.812146 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:24.812150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:24.812163 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:16:24.812288 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:24.812291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:24.812410 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:24.812510 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:24.812710 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.813239 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.814026 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.814286 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.814376 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:24.814638 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:24.815074 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:16:24.815165 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:24.815222 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 16:16:24.815269 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260924 16:16:24.822580 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:24.818585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.1.253.124:43251)
0924 16:16:24.818690 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:24.818702 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:24.819099 (+ 397us) server_negotiation.cc:407] Connection header received
0924 16:16:24.819177 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:24.819180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:24.819227 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:24.819296 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:24.819696 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.820489 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.821470 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.821691 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.821818 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:24.822037 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:24.822118 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:16:24.822184 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:24.822369 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 16:16:24.822416 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 16:16:24.823182 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:24.818900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.1.253.125:45175)
0924 16:16:24.819033 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:24.819037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:24.819296 (+ 259us) server_negotiation.cc:407] Connection header received
0924 16:16:24.819422 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:24.819425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:24.819484 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:24.819563 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:24.819900 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.820447 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.821758 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.822079 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.822190 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:24.822633 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:24.822728 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:16:24.822783 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:24.822973 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 16:16:24.823033 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:16:24.823606 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:24.818519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.1.253.126:42401)
0924 16:16:24.818757 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:24.818761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:24.819081 (+ 320us) server_negotiation.cc:407] Connection header received
0924 16:16:24.819185 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:24.819188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:24.819340 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:24.819489 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:24.819760 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.820568 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.821778 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.822119 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.822240 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:24.822708 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:24.823052 (+ 344us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:16:24.823138 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:24.823391 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 16:16:24.823465 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":62,"threads_started":1}
I20260924 16:16:24.826716 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 24949
I20260924 16:16:24.834657 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 25026
I20260924 16:16:24.841661 2037 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pu02A/build/debug/bin/kudu with pid 25101
2026-09-24T16:16:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3169 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6324 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (287588 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST